Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add snowflake_put output #1160

Merged
merged 3 commits into from
Mar 31, 2022

Conversation

mihaitodor
Copy link
Collaborator

@mihaitodor mihaitodor commented Mar 8, 2022

For anyone who'd like to test this out, Snowflake allows you to create a 30-day free trial account here.

I'd like to add a bunch of lints to the config, but wasn't sure what's the best way to do that currently. I know I did it here, but not sure how to get to that when starting from service.NewConfigSpec().

TODO: Add lints for:

  • upload_parallel_threads only allows values between 1 and 99 (see here)
  • password / SSH private key are exclusive
  • don't allow snowpipe to be set when in user/password auth mode

Note: This is part of #658.

@mihaitodor mihaitodor requested a review from Jeffail March 8, 2022 22:38
@mihaitodor mihaitodor force-pushed the add-snowflake-put-output branch 4 times, most recently from 8bf2d44 to 6d531df Compare March 10, 2022 16:07
@mihaitodor mihaitodor force-pushed the add-snowflake-put-output branch 3 times, most recently from 1d161ca to 93868bb Compare March 23, 2022 21:40
@Jeffail
Copy link
Collaborator

Jeffail commented Mar 30, 2022

@mihaitodor I've added blobl linting: e7ee59a, worth trying when you get a chance

@mihaitodor mihaitodor force-pushed the add-snowflake-put-output branch from 93868bb to ff93f9b Compare March 31, 2022 14:23
@mihaitodor
Copy link
Collaborator Author

@Jeffail Thanks so much for adding the linting functionality! It's awesome, but I found it useful to also add a LintRule method for the ConfigSpec.

In terms of ergonomics, it might be handy to allow users to chain multiple LintRules, so then each of them is run and they can see each separate lint if there are multiple issues. The match I used works too, but they'll only get the first error it encounters. Guess it can be improved with a fancier blobalng query, but that's harder to maintain.

@mihaitodor mihaitodor force-pushed the add-snowflake-put-output branch from ff93f9b to a52c748 Compare March 31, 2022 14:44
@mihaitodor mihaitodor force-pushed the add-snowflake-put-output branch 2 times, most recently from af91f09 to 1df32e2 Compare March 31, 2022 14:53
@mihaitodor mihaitodor force-pushed the add-snowflake-put-output branch from 1df32e2 to 8c7fb13 Compare March 31, 2022 14:56
@Jeffail
Copy link
Collaborator

Jeffail commented Mar 31, 2022

Nice, extra props for fixing the batch docs.

@Jeffail Jeffail merged commit 453ab82 into redpanda-data:main Mar 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants