diff --git a/.gitignore b/.gitignore index 085f7740..5383814d 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,6 @@ result-* # generic dist/ + +# Go +go.work \ No newline at end of file diff --git a/config/.gitkeep b/config/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/docs/README.md b/docs/README.md index f19c8705..add22bea 100644 --- a/docs/README.md +++ b/docs/README.md @@ -31,9 +31,14 @@ The operator creates and manages all necessary Kubernetes resources for these co ``` multigres-operator/ -├── api/v1alpha1/ # CRD definitions +├── go.mod # Root module ├── cmd/multigres-operator/ # Main entry point -└── internal/ # Controller and resource builders +└── pkg/ # Multi-module structure + ├── cluster-handler/ # Cluster orchestration (separate module) + ├── data-handler/ # Data plane management (separate module) + └── resource-handler/ # Component resources (separate module) + +Note: go.work can be created locally for development. It is in .gitignore ``` ### Documentation and Configuration diff --git a/docs/architecture.md b/docs/architecture.md index fb8d922c..86cd1f4e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -40,36 +40,68 @@ ``` multigres-operator/ -├── api/v1alpha1/ # CRD definitions and types +├── go.mod # Root module (primarily for cmd) ├── cmd/multigres-operator/ # Main entry point -├── internal/ -│ ├── controller/ # Main reconciler and component reconcilers -│ │ ├── etcd/ -│ │ ├── multigateway/ -│ │ ├── multiorch/ -│ │ └── multipooler/ -│ ├── resources/ # Pure resource builder functions -│ ├── webhook/ # Admission webhooks (defaulting, validation) -│ └── testutil/ # Test helpers and utilities +├── pkg/ +│ ├── cluster-handler/ # Cluster-level orchestration (separate Go module) +│ │ ├── go.mod +│ │ └── controller/ +│ │ └── multigrescluster/ # MultigresCluster reconciler +│ ├── data-handler/ # Data plane management (separate Go module) +│ │ ├── go.mod +│ │ └── controller/ +│ │ └── cell/ # Cell reconciler +│ └── resource-handler/ # Component resources (separate Go module) +│ ├── go.mod +│ └── controller/ +│ ├── multigateway/ +│ ├── multiorch/ +│ ├── multipooler/ +│ └── etcd/ ├── config/ # Kubernetes manifests for operator deployment ├── docs/ # Architecture, conventions, and development guides └── plans/ # Planning documents -``` - -### API Layer (`api/v1alpha1`) -- **Purpose**: Defines Multigres custom resource schema -- **Components**: MultigresSpec, MultigresStatus, component specs (MultiGatewaySpec, MultiOrchSpec, MultiPoolerSpec, EtcdSpec) -- **Validation**: Kubebuilder markers for OpenAPI validation and defaults -### Controller Layer (`internal/controller`) -- **MultigresReconciler**: Main controller - manages lifecycle, finalizers, status aggregation -- **Component Reconcilers**: etcd, multigateway, multiorch, multipooler reconcilers run in parallel -- **Responsibilities**: Create/update resources, check component health, update status +Note: go.work file can be created locally for development convenience. It is in .gitignore. +``` -### Resource Layer (`internal/resources`) -- **Pure Functions**: Build Kubernetes manifests (Deployments, StatefulSets, Services, HPAs) -- **Label Management**: Consistent label generation for resource selection -- **No Side Effects**: Same input always produces same output +### Multi-Module Architecture + +The operator is organized into three independent Go modules with clear separation of concerns. Developers can use **Go workspaces** locally (`go.work`) for easier cross-module development, though this file is not committed to the repository: + +#### Cluster Handler Module (`pkg/cluster-handler`) +- **Purpose**: High-level cluster orchestration +- **Responsibilities**: + - Uses the `MultigresCluster` custom resource as its primary input + - Coordinates creation of child resources (MultiGateway, MultiOrch, MultiPooler, Etcd, Cell) + - Aggregates status from child resources + - Handles cluster-wide lifecycle (finalizers, upgrades) +- **Independence**: Can be tested and versioned separately from resource handlers + +#### Data Handler Module (`pkg/data-handler`) +- **Purpose**: Data plane management and topology +- **Responsibilities**: + - Uses the `Cell` custom resource as its primary input (data topology abstraction) + - Handles data plane configuration and routing + - Coordinates with Vitess/Multigres data components +- **Independence**: Data plane logic isolated from control plane and resource management + +#### Resource Handler Module (`pkg/resource-handler`) +- **Purpose**: Individual component resource management +- **Responsibilities**: + - Uses the `MultiGateway`, `MultiOrch`, `MultiPooler`, `Etcd` custom resources as its primary inputs + - Creates and reconciles Kubernetes resources (Deployments, StatefulSets, Services, HPAs) + - Implements pure resource builder functions + - Component-specific health checking and status reporting +- **Independence**: Component reconcilers can evolve independently of cluster orchestration + +### Benefits of Multi-Module Structure + +- **Clear Boundaries**: Each module has distinct responsibilities and can be understood independently +- **Independent Testing**: Unit and integration tests scoped to each module +- **Parallel Development**: Teams can work on different modules without conflicts +- **Selective Imports**: Main binary only imports what it needs from each module +- **Version Flexibility**: Modules can evolve at different paces (though currently developed in lockstep) ## Core Components @@ -85,14 +117,14 @@ The operator follows a standard Kubernetes reconciliation pattern: ### Component Reconcilers -Each Multigres component has its own reconciler: +The **resource-handler** module contains individual reconcilers for each Multigres component: -- **etcd Reconciler**: Manages StatefulSet for etcd cluster, headless and client Services -- **multigateway Reconciler**: Manages Deployment, Service, and optional HPA for MultiGateway -- **multiorch Reconciler**: Manages Deployment and optional HPA for MultiOrch -- **multipooler Reconciler**: Manages StatefulSet with multi-container pods (pooler, pgctld, postgres), and optional HPA +- **etcd Reconciler** (`pkg/resource-handler/controller/etcd`): Manages StatefulSet for etcd cluster, headless and client Services +- **multigateway Reconciler** (`pkg/resource-handler/controller/multigateway`): Manages Deployment, Service, and optional HPA for MultiGateway +- **multiorch Reconciler** (`pkg/resource-handler/controller/multiorch`): Manages Deployment and optional HPA for MultiOrch +- **multipooler Reconciler** (`pkg/resource-handler/controller/multipooler`): Manages StatefulSet with multi-container pods (pooler, pgctld, postgres), and optional HPA -All component reconcilers run in parallel. +All component reconcilers run in parallel and are independent of each other. ### Resource Builders @@ -146,8 +178,8 @@ Current preference is init container pattern (same as Istio, NGINX Ingress), but ## Technology Stack ### Language and Runtime -- **Language**: Go 1.24+ -- **Key Features**: Concurrency, interfaces, strong typing +- **Language**: Go 1.25+ +- **Key Features**: Concurrency, interfaces, strong typing, workspaces for multi-module projects ### Key Dependencies - **Framework**: Kubebuilder v3 - scaffolding and patterns for Kubernetes operators diff --git a/docs/implementation-guide.md b/docs/implementation-guide.md index 66d998fb..9359729a 100644 --- a/docs/implementation-guide.md +++ b/docs/implementation-guide.md @@ -5,13 +5,13 @@ ## Development Environment ### Language Configuration -- **Language**: Go 1.24+ -- **Build System**: Make for task orchestration, Go modules for dependencies -- **Project Structure**: Standard Kubebuilder layout - `api/`, `cmd/`, `internal/`, `config/` +- **Language**: Go 1.25+ +- **Build System**: Make for task orchestration, Go workspaces for multi-module management +- **Project Structure**: Multi-module workspace - `go.work`, `pkg/{module}/`, `cmd/`, `config/` - **Linting**: golangci-lint (configured in `.golangci.yml`) ### Required Tools -- **go**: 1.24 or higher +- **go**: 1.25 or higher (workspace support required) - **kubectl**: Kubernetes CLI for cluster interaction - **kind**: Local Kubernetes cluster for testing (or other local cluster) - **kubebuilder**: Optional - for regenerating CRDs and scaffolding @@ -22,10 +22,14 @@ - **kustomize**: Included with kubectl 1.14+, used for manifest management ### Dependency Management -- **Go Modules**: `go.mod` and `go.sum` for dependency tracking -- **Version Pinning**: Pin controller-runtime and client-go to compatible versions -- **Vendor Directory**: Not used - rely on module cache -- **Updating Dependencies**: Use `go get -u` carefully, test thoroughly after updates +- **Per-Module Dependencies**: Each module (`cluster-handler`, `data-handler`, `resource-handler`) has its own `go.mod` and `go.sum` +- **Version Pinning**: Pin controller-runtime and client-go to compatible versions in each module +- **Go Workspaces (Local Only)**: + - Create `go.work` locally for easier multi-module development: `go work init ./pkg/cluster-handler ./pkg/data-handler ./pkg/resource-handler` + - `go.work` is in `.gitignore` + - **IMPORTANT**: `go.work` MUST NOT be committed - committing it would break the build and cause artifacts to ignore pinned versions + - Workspaces simplify local development by allowing cross-module references without publishing +- **Updating Dependencies**: Update in each module separately with `go get -u`, test thoroughly after updates ## Coding Standards @@ -64,15 +68,17 @@ Common markers: ### Testing Strategy -**Unit Tests** (internal/resources, internal/webhook): +**Unit Tests** (per module): - Test pure resource builder functions with table-driven tests - Mock nothing - builders are pure functions - Focus on correct Kubernetes manifest generation +- Run tests in each module: `cd pkg/resource-handler && go test ./...` -**Integration Tests** (internal/controller): +**Integration Tests** (per module): - Use `envtest` - provides real Kubernetes API without full cluster - Test reconciliation loops end-to-end - Verify resource creation, updates, and status updates +- Each module has its own integration tests scoped to its controllers **Test Commands**: ```bash @@ -145,12 +151,13 @@ make manifests make install ``` -**Controller Changes** (`internal/controller/`): -- Run `make test` frequently during development +**Controller Changes** (`pkg/{module}/controller/`): +- Run tests in the specific module: `cd pkg/resource-handler && go test ./...` - Integration tests catch reconciliation bugs early - Use `make run` for quick iteration (no docker build needed) +- Changes in one module don't require rebuilding others during local development -**Resource Builder Changes** (`internal/resources/`): -- Write/update table-driven tests first -- Run `make test` - very fast feedback loop -- Integration tests verify end-to-end behavior \ No newline at end of file +**Module-Specific Development**: +- **Resource Handler** (`pkg/resource-handler`): Component reconcilers and resource builders +- **Cluster Handler** (`pkg/cluster-handler`): MultigresCluster orchestration logic +- **Data Handler** (`pkg/data-handler`): Cell management and data plane configuration \ No newline at end of file diff --git a/internal/.gitkeep b/internal/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/pkg/cluster-handler/controller/multigrescluster/dummy.go b/pkg/cluster-handler/controller/multigrescluster/dummy.go new file mode 100644 index 00000000..f8681eea --- /dev/null +++ b/pkg/cluster-handler/controller/multigrescluster/dummy.go @@ -0,0 +1,5 @@ +package multigrescluster + +func Dummy() string { + return "dummy string from cluster-handler's multigrescluster controller" +} diff --git a/pkg/cluster-handler/go.mod b/pkg/cluster-handler/go.mod new file mode 100644 index 00000000..be5c4f59 --- /dev/null +++ b/pkg/cluster-handler/go.mod @@ -0,0 +1,3 @@ +module github.com/numtide/multigres-operator/pkg/cluster-handler + +go 1.25.0 diff --git a/pkg/data-handler/controller/cell/dummy.go b/pkg/data-handler/controller/cell/dummy.go new file mode 100644 index 00000000..4f4466c2 --- /dev/null +++ b/pkg/data-handler/controller/cell/dummy.go @@ -0,0 +1,5 @@ +package cell + +func Dummy() string { + return "dummy string from data-handler's cell controller" +} diff --git a/pkg/data-handler/go.mod b/pkg/data-handler/go.mod new file mode 100644 index 00000000..42261bf6 --- /dev/null +++ b/pkg/data-handler/go.mod @@ -0,0 +1,3 @@ +module github.com/numtide/multigres-operator/pkg/data-handler + +go 1.25.0 diff --git a/pkg/resource-handler/controller/multigateway/dummy.go b/pkg/resource-handler/controller/multigateway/dummy.go new file mode 100644 index 00000000..0560d57d --- /dev/null +++ b/pkg/resource-handler/controller/multigateway/dummy.go @@ -0,0 +1,5 @@ +package multigateway + +func Dummy() string { + return "dummy string from resource-handler's multigateway controller" +} diff --git a/pkg/resource-handler/go.mod b/pkg/resource-handler/go.mod new file mode 100644 index 00000000..370abd64 --- /dev/null +++ b/pkg/resource-handler/go.mod @@ -0,0 +1,3 @@ +module github.com/numtide/multigres-operator/pkg/resource-handler + +go 1.25.0 diff --git a/plans/.gitkeep b/plans/.gitkeep new file mode 100644 index 00000000..e69de29b