Skip to content

Add functions parameter to template transformers for reusable transformer function definitionsΒ #544

@dp-dandy

Description

@dp-dandy

Is your feature request related to a problem? Please describe.

Currently, template transformers (JSON, hstore, and template) are limited to the functions available through greenmasktoolkit.FuncMap() and sprig.FuncMap(). Users cannot leverage the full catalog of pgstream transformers (like neosync_firstname, greenmask_choice, etc.) within templates, creating inconsistency when trying to apply the same transformation logic across different data structures.

This is particularly problematic for entity-event patterns where you need to apply the same transformation to both:

  1. A column in an entity table (using dedicated transformer)
  2. A field within a JSON column in an events table (currently limited to basic template functions)

Example of what doesn't work today:

- schema: schema_a
  table: entity_a
  column_transformers:
    column_a:
      name: neosync_firstname  # Works fine
      parameters:
        preserve_length: true
        max_length: 100

- schema: schema_a
  table: events
  column_transformers:
    json_b_column:
      name: json
      parameters:
        operations:
          - operation: set
            path: "column_a"
            value_template: '{{ ??? }}'  # No equivalent function available

Describe the solution you'd like

Add a functions parameter to template-based transformers (JSON, hstore, and template) that allows users to define reusable transformer functions with their full parameter sets, then reference those functions by name in templates.

Proposed Configuration Structure:

- schema: schema_a
  table: events
  column_transformers:
    json_b_column:
      name: json
      parameters:
        functions:
          my_firstname_transform:
            name: neosync_firstname
            parameters:
              preserve_length: true
              max_length: 100
          my_email_transform:
            name: neosync_email
            parameters:
              preserve_length: true
              preserve_domain: true
        operations:
          - operation: set
            path: "user.first_name"
            value_template: '{{ my_firstname_transform .GetValue }}'
          - operation: set
            path: "user.email"
            value_template: '{{ my_email_transform .GetValue }}'

Describe alternatives you've considered

  1. Inline function parameters in templates: {{ neosyncFirstname 100 true }} - but this makes templates complex and loses type safety
  2. Global transformer function registry: Expose all transformers as template functions - but this pollutes the template namespace and loses parameter control
  3. Nested transformer configuration: Allow transformers within transformers - but this creates complex nested configuration

Additional context

Benefits of this approach:

  • Reusability: Define transformer functions once, use multiple times
  • Full parameter support: Access to all transformer parameters and validation
  • Clean separation: Configuration stays in YAML, templates remain simple
  • Consistency: Same transformation logic works across entity tables and JSON fields
  • Type safety: Parameters validated through existing transformer system
  • Backward compatibility: Existing configurations continue to work

Affected transformers:

  • json transformer (pkg/transformers/json_transformer.go)
  • hstore transformer (pkg/transformers/hstore_transformer.go)
  • template transformer (pkg/transformers/template_transformer.go)

This enhancement would enable consistent data masking across entity tables and their corresponding event logs while leveraging the full power of pgstream's transformer ecosystem within structured data transformations.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesttransformersTransformer related work

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions