Plugin Framework

[jlantz]

CumulusCI has always had an all batteries included approach to package dependencies. Installing the cumulusci package pulls in a lot of Python packages, many of which are used for more edge case functionality.

There are also plenty of existing Python libraries which would be great to use in expanding CumulusCI. The problem is adding more dependencies to core doesn't seem like the right thing anymore. Core is already too bloated as is.

A few years ago I prototyped a really simple plugin framework for CumulusCI using the following concepts:

1. Do not handle install/uninstall of plugins. Plugins are expected to already be installed in the runtime
2. Projects can require plugins in the new top level plugins key in cumulusci.yml containing a list of Python package names
3. Plugins are loaded at project config initialization and injected into the configuration dictionary merge stack in order between the universal and project config merge.

This allows projects full flexibility to require a plugin be installed in the runtime with no impact on projects not using plugins.

Advantages

• Support third party extensions with additional Python dependencies like the Stripe Python SDK or OpenAI clients
• Create a framework for moving functionality out of cumulusci into separate packages that can be optionally installed
• Allow easier community sharing new tasks and flows without the burden of full contribution back to CumulusCI core

Implementation

The implementation of all of this is amazingly only ~30 lines of code changes to cumulusci.core.config.project_config.BaseProjectConfig._load_config as seen here:
https://github.com/SFDO-Tooling/CumulusCI/compare/main...muselab-d2x:CumulusCI:feature/plugins?diff=split&w=#diff-1f678872ff8728a8dab42acfb218e9f83932f3a91af85e7c63494298e3e65657R142-R226

I'll work on test cases and get a PR for this implementation but figured I'd kick off the new discussions with a post to see if anyone has any thoughts on the idea.

[jstvz]

Thanks for opening our first discussion with your proposal. I especially appreciate the simple implementation. Adding a plugin architecture for CumulusCI to support custom services and tasks would be a big improvement to the framework. I also agree that the batteries-included approach has both benefits and drawbacks.

Context

I reviewed a write-up David Reed did last year, which stressed the need for better organization of tasks and flows. For example, Stewart suggested adding an imports section to the cumulusci.yml file to bring in shared tasks and flows from other projects. This could help keep the main configuration clean and makes it easier for teams to maintain consistency without duplicating work.

Additionally, your recent contributions pointed out that managing third-party dependencies adds administrative overhead and slows down the review process.

Here were the main ideas we discussed internally around that time:

• Organizing Configuration: Add an imports section to keep project settings separate from shared tasks and flows, to reduce clutter.
• Code Reuse Across Projects: Allow projects to import configurations from shared repositories, promoting code reuse.
• Custom Python Tasks: Enable custom Python tasks to work alongside YAML tasks for better integration.
• Enhanced Dependency Management: Use a plugin_packages key to check for required Python packages at runtime, to reduce issues with missing dependencies.

Approaches We Are Considering

These are some broad strokes:

• Import Configuration Files: Allow projects to import shared tasks and flows from a separate folder or repository. This helps maintain a clean configuration and reduces the need to update multiple files across different projects.
• Plugin Management: Use namespace packages installed via pip that implement one or more hooks. For example, a system like Pluggy can help add CLI commands in a way that works well with existing features (a practical example is in the pluggable CLI example repository). This ensures that needed components are ready before running tasks.
• Custom Code Integration: Plugins could include custom Python code that works with defined YAML tasks and flows. This integration improves flexibility and makes it easy to add new features(e.g. hashing) without changing the core system.

Questions for Discussion

• How many types of hooks (i.e., commands, tasks, flows, plans, and services) do we really need? How much can we do incrementally?
• What's the right way to handle missing plugin installations for the best UX? Should we aim for a SFDX-style representation of installed plugins?
• What challenges might we face with plugin dependencies?
• Should we keep core functions separate from optional plugins to avoid confusion? (Can we?)
• Do we need a versioning strategy for plugins to ensure they work well with CumulusCI?