Checklist
kargo version
Client Version: v1.8.4
Server Version: v1.8.1
Proposed Feature
Add Warehouse filtering on includePaths to webhook receivers for ClusterConfig webhooks.
Motivation
We store our application configuration in a directory called apps/ in a monorepo. We have plans to migrate more applications to this repository and might arrive at 150+ applications. Currently we have two warehouse configured:
- ECR Warehouse (not relevant per this discussion)
- Git warehouse
spec:
subscriptions:
- git:
repoURL: https://github.com/<owner>/<repo>.git
commitSelectionStrategy: NewestFromBranch
includePaths:
- apps/foo
According to the documentation here,
A webhook receiver's only job is to extract a repository URL from the webhook request's payload, query for all Warehouse resources across all Projects having subscriptions to that repository, and request each to execute their artifact discovery process.
This means that ALL Warehouses subscribed to this repository will get triggered to perform an update.
I would like to see filtering added to the Webhook receiver to notify only specific Warehouses that match the correct includePaths from the payload.
I checked that the Github push event does include the files:
type PushEvent struct {
// ... other fields
HeadCommit *HeadCommit `json:"head_commit,omitempty"`
Commits []*HeadCommit `json:"commits,omitempty"`
}
type HeadCommit struct {
Added []string `json:"added,omitempty"`
Removed []string `json:"removed,omitempty"`
Modified []string `json:"modified,omitempty"`
// ... other fields
}
Work would entail:
- Extract file paths from the webhook payload (github.go:215-225)
case *gh.PushEvent:
repoURLs = []string{urls.NormalizeGit(e.GetRepo().GetCloneURL())}
ref := e.GetRef()
// NEW: Extract file paths from HeadCommit
var filePaths []string
if e.HeadCommit != nil {
filePaths = append(filePaths, e.HeadCommit.Added...)
filePaths = append(filePaths, e.HeadCommit.Modified...)
filePaths = append(filePaths, e.HeadCommit.Removed...)
}
- Update
shouldRefresh in (refresh.go 148-179)
- add file path matching
- check if modified files match
includePaths
- respect the
excludePaths
- Pass file paths through call chain:
- update
refreshWarehouses signature to accept file paths
- update
shouldRefresh to evaluate path patterns
- Add documentation
This would greatly reduce the Warehouse refresh.
The only other way I can think of tackling this is to cache the github repository and use that across multiple warehouses so that the Github API queries are reduced and Warehouses that share a repository can detect changes instantly.
Checklist
kargo version, if applicable.Proposed Feature
Add Warehouse filtering on
includePathsto webhook receivers for ClusterConfig webhooks.Motivation
We store our application configuration in a directory called
apps/in a monorepo. We have plans to migrate more applications to this repository and might arrive at 150+ applications. Currently we have two warehouse configured:According to the documentation here,
This means that ALL Warehouses subscribed to this repository will get triggered to perform an update.
I would like to see filtering added to the Webhook receiver to notify only specific Warehouses that match the correct
includePathsfrom the payload.I checked that the Github
pushevent does include the files:Work would entail:
shouldRefreshin (refresh.go 148-179)includePathsexcludePathsrefreshWarehousessignature to accept file pathsshouldRefreshto evaluate path patternsThis would greatly reduce the Warehouse refresh.
The only other way I can think of tackling this is to cache the github repository and use that across multiple warehouses so that the Github API queries are reduced and Warehouses that share a repository can detect changes instantly.