Discussion: On the Possibility of Changing Maxwell's Code Organization

[guxingke]

Background

Maxwell is an excellent tool for capturing and distributing MySQL binlog changes. Currently, Maxwell's core logic and various producers are tightly coupled within a single codebase. While it is technically possible to integrate Maxwell as a library, the current monolithic package pulls in a large number of unnecessary dependencies, which increases the complexity of integration and maintenance. This forces any project that wants to use its core functionality to either run a full Maxwell daemon or integrate a bulky library with redundant features.

An Idea

I have an idea about adjusting Maxwell's code organization and would like to consult and discuss its feasibility with the project maintainers.

My idea is to restructure the Maxwell codebase into a modular architecture without changing the current distribution method. Specifically, we could:

1. Create a maxwell-core module: This module would contain only the code for binlog parsing, change capture, and core routing logic. It would be very lean and would not include any specific producer implementations.
2. Create separate producer modules: For example, maxwell-producer-kafka, maxwell-producer-rabbitmq, etc. These modules would depend on maxwell-core.

The Benefits of This Change

• Lighter and more focused: The maxwell-core module would focus solely on its core tasks. When used as a library, it would no longer pass on unnecessary dependencies, greatly reducing project complexity.
• Greater flexibility: Other projects could easily integrate the streamlined maxwell-core into their own code and directly call its core functions without needing to run a separate Maxwell daemon.
• Encourages innovation: The community could more easily develop and contribute new producer modules or create custom data processing logic without modifying the core codebase.

I understand this adjustment might require some effort, but in the long run, it would significantly improve Maxwell's flexibility and extensibility.

I'm curious to hear your thoughts on this and whether such a change is on the roadmap for the future. I look forward to your reply.

[osheroff]

I like it, especially if it can be done without too much source code reorganizing. But someone (you?) would need to take ownership over the refactor

[guxingke]

Thank you for the reply and for your positive feedback on the idea! I'm glad to hear you're open to it.

I'm happy to take ownership of this refactoring work. I agree that this direction is beneficial for the project's long-term health and flexibility.

After some initial thought, here is my preliminary plan for the refactor. The goal is to convert the project to a multi-module structure without affecting the current distribution methods (tar.gz, Maven Central).

---

Preliminary Refactoring Plan

The approach is to separate the core logic from the final application assembly.

1. New maxwell-parent Module: This will be the new root of the project, defining common build settings and dependencies.
2. New maxwell-core Module: This will contain the core source code (src/main/java, src/main/antlr4, etc.). A key change here is that specific producer dependencies (e.g., Kafka) will be configured with a provided scope. This ensures they are not transitively included when maxwell-core is used as a standalone library, keeping the core module lean.
3. New maxwell Module: This module will contain the application-specific resources for the daemon (src/main/assemble, src/main/resources/log4j2.xml, src/example). This module will depend on maxwell-core, and its POM file will explicitly include all producer dependencies with a compile scope (the default).

This structure ensures that the final .tar.gz and Maven Central artifacts remain fully functional while making the maxwell-core library clean and easy to integrate for other projects.

My next step will be to implement this plan in a new branch and create a Draft Pull Request. This will allow us to discuss the implementation details with actual code. I'll share the link here once it's ready.