Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically fill fields from input step at runtime when the fields are not specified #21

Open
juarezr opened this issue Oct 3, 2017 · 0 comments

Comments

@juarezr
Copy link

juarezr commented Oct 3, 2017

  • Uses cases:
    1. use the same transformation for exporting for diferent sources
    2. switching between output formats according the needs (eg: avro, csv, parquet)
  • Reproducing the concept:
    1. Create a extract transformation with a final step of type 'Copy rows to result'
    2. Create a output transformation with a initial step of type 'Get rows from result'
    3. Connect the initial step to a second step of type 'Avro Output'
    4. Create a Job and connect the extract transformation to the output transformation
    5. Today the Avro Output plugin create a Avro file with zero columns.
    6. It should fill the columns at runtime inspecting the columns in the previous step
  • Problems:
    • If the Fields are not specified in the fields tab, Avro Output plugin creates a file with no columns, even the the previous step has fields defined at runtime.
    • If the Avro Output step are replaced by a 'Text file output', the resulting file is created with all columns correct.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant