Split up generate_data and add a mix_datasets top level API

[bbrowning]

This separates out instructlab.sdg.generate_data into separate functions, where generate_data just calls into these separate functions instead of containing all the logic in itself directly.

This enables us to expose a new top-level API, mix_datasets, to allow users to mix datasets. It also lays the groundwork for how we might split some of the pre-processing and post-processing out into separate modules or other repositories, as now there's a clean separation between the various concerns that were previously all intertwined within generate_data.

[bbrowning]

@jwm4 Does something like this help with some of your needs? A top-level API to only do the preprocessing steps but not any data generation?

[cdoern]

This adds a simple CLI usable within SDG itself to do all taxonomy preprocessing via:
python -m instructlab.sdg.cli.taxonomy_to_samples --taxonomy-path /path/to/my/taxonomy/ --output-dir /path/to/my/output/samples --log-level DEBUG

Does this confuse the user story a bit of what we expect to be a "user facing" entry point to InstructLab? or is this CLI just meant to be a sort of dev environment for quick use? If its the later, then this makes sense.

However, I do think it would make sense to instead focus on adding this CLI functionality into an ilab data process command. That could be as simple now as adding the proper plumbing to call whatever whatever API is being invoked via the new CLI being introduced here.

With changes like this that expose a new user entry point we risk divergence and decreased functionality in ilab due to new usage patterns that the dev team might more frequently use rather than what everyday users will be running

[cdoern]

one comment related to the above

[bbrowning]

I believe this is ready for larger review. Some particular things I'd like to call out for reviewers to think about:

The only other top-level API we expose here via __init__.py is mix_datasets. Does the Python API of mix_datasets look reasonable? This is one we'd want to support for advanced users that need to adjust how generated data gets mixed.

The old monolithic generate_data was split into preprocess_taxonomy, generate_taxonomy, generate_taxonomy_eval, postprocess_taxonomy, and mix_datasets. Are these a reasonable level of granularity, at least in a first pass of splitting things up? For backwards-compatibility we retained a generate_data method that just calls all of the above in turn to keep the same behavior as before. None of these except mix_datasets are exposed as top-level importable functions from instructlab-sdg. So, we can iterate on naming and further splitting as needed in future releases unless there's strong disagreement here.

We now write more intermediate files to disk between stages, including additional metadata (ie columns) in these jsonl files to keep track of things like whether a sample generated from the taxonomy was a knowledge, freeform, or grounded skill as we use that to differentiate which pipeline to run. Are the leaf_node_type and leaf_node_path column names reasonable to write into our sample files for this purpose?

Lastly, we now do all the preprocessing for all leaf nodes before moving on to generation. We do all generation for all leaf nodes before moving on to postprocessing. Previously, we operated on a single leaf node at a time end-to-end. In other words, previously we'd preprocess, generate, and postprocess leaf node A. Then we'd do the same for leaf node B. Now, we preprocess leaf nodes A & B, then generate for A & B, and finally postprocess A & B. This was required for decoupling things, and sets us up for some future scenarios where we'll need to entirely finish one stage of our processing before moving on to the next because some stages may use different models, such as subset selection with an embedding model. Any concerns about moving to this approach?

Obviously I'd love feedback for any/all parts of the PR, but just calling out the above as some places to spend a bit more focused time during a review.

[aakankshaduggal]

Overall lgtm, great work here @bbrowning
I have minor comment, but we don't have to get it in with this PR.

[bbrowning]

I like the suggestion @aakankshaduggal , so incorporating that directly here. Thanks!

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. @bbrowning please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[bbrowning]

Rebased this to fix a merge conflict, added a few examples of using the mixing API, along with a test to ensure those examples work as expected.

[cdoern]

approving from a CLI point of view. We can iterate on the public APIs and when they should move to core.