Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Clarification of role in the pipeline #165

Open
olorin opened this issue Jun 3, 2016 · 0 comments
Open

Clarification of role in the pipeline #165

olorin opened this issue Jun 3, 2016 · 0 comments

Comments

@olorin
Copy link
Contributor

olorin commented Jun 3, 2016

Recent clarifications of the role of warden within the pipeline:

  • warden doesn't necessarily make the call on "dataset is too broken, stop processing"
  • tool which does this uses metadata from warden (number of borked rows et cetera)
  • need to add some functionality to warden regarding line numbers and counts and such
  • then think about/build this other tool
  • for now, warden checks failing (i.e., detecting any bad data) shouldn't cause a job failure
  • there will be exceptions to the above, e.g., differing field counts or maybe PII
olorin added a commit that referenced this issue Feb 24, 2017
@olorin
Default to exiting with success if the check run is successful
e1092f8 
Towards clarification of warden's place in the pipeline in #165 -
warden shouldn't be making the "go/no-go" call on the dataset itself
but should be generating metadata such that something else can make
the decision. As part of this, we don't want jobs to fail/abort with
error if the checks in fact ran successfully.
@olorin olorin mentioned this issue Feb 24, 2017
Merged
olorin added a commit that referenced this issue Feb 24, 2017
@olorin
Default to exiting with success if the check run is successful
d13ec48 
Towards clarification of warden's place in the pipeline in #165 -
warden shouldn't be making the "go/no-go" call on the dataset itself
but should be generating metadata such that something else can make
the decision. As part of this, we don't want jobs to fail/abort with
error if the checks in fact ran successfully.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@olorin