-
Notifications
You must be signed in to change notification settings - Fork 35
Description
Is your feature request related to a problem? Please describe.
Currently, template transformers (JSON, hstore, and template) are limited to the functions available through greenmasktoolkit.FuncMap() and sprig.FuncMap(). Users cannot leverage the full catalog of pgstream transformers (like neosync_firstname, greenmask_choice, etc.) within templates, creating inconsistency when trying to apply the same transformation logic across different data structures.
This is particularly problematic for entity-event patterns where you need to apply the same transformation to both:
- A column in an entity table (using dedicated transformer)
- A field within a JSON column in an events table (currently limited to basic template functions)
Example of what doesn't work today:
- schema: schema_a
table: entity_a
column_transformers:
column_a:
name: neosync_firstname # Works fine
parameters:
preserve_length: true
max_length: 100
- schema: schema_a
table: events
column_transformers:
json_b_column:
name: json
parameters:
operations:
- operation: set
path: "column_a"
value_template: '{{ ??? }}' # No equivalent function availableDescribe the solution you'd like
Add a functions parameter to template-based transformers (JSON, hstore, and template) that allows users to define reusable transformer functions with their full parameter sets, then reference those functions by name in templates.
Proposed Configuration Structure:
- schema: schema_a
table: events
column_transformers:
json_b_column:
name: json
parameters:
functions:
my_firstname_transform:
name: neosync_firstname
parameters:
preserve_length: true
max_length: 100
my_email_transform:
name: neosync_email
parameters:
preserve_length: true
preserve_domain: true
operations:
- operation: set
path: "user.first_name"
value_template: '{{ my_firstname_transform .GetValue }}'
- operation: set
path: "user.email"
value_template: '{{ my_email_transform .GetValue }}'Describe alternatives you've considered
- Inline function parameters in templates:
{{ neosyncFirstname 100 true }}- but this makes templates complex and loses type safety - Global transformer function registry: Expose all transformers as template functions - but this pollutes the template namespace and loses parameter control
- Nested transformer configuration: Allow transformers within transformers - but this creates complex nested configuration
Additional context
Benefits of this approach:
- Reusability: Define transformer functions once, use multiple times
- Full parameter support: Access to all transformer parameters and validation
- Clean separation: Configuration stays in YAML, templates remain simple
- Consistency: Same transformation logic works across entity tables and JSON fields
- Type safety: Parameters validated through existing transformer system
- Backward compatibility: Existing configurations continue to work
Affected transformers:
jsontransformer (pkg/transformers/json_transformer.go)hstoretransformer (pkg/transformers/hstore_transformer.go)templatetransformer (pkg/transformers/template_transformer.go)
This enhancement would enable consistent data masking across entity tables and their corresponding event logs while leveraging the full power of pgstream's transformer ecosystem within structured data transformations.