Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run delete/abort fails when Polygraph is in some broken states #6

Open
LionsPhil opened this issue Jun 14, 2016 · 0 comments
Open

Run delete/abort fails when Polygraph is in some broken states #6

LionsPhil opened this issue Jun 14, 2016 · 0 comments
Labels

Comments

@LionsPhil
Copy link
Owner

Plumage does not recover well from some inconsistent states where runs have failed, because attempts to clean up across all the machines involved become fatal. For example, if there's a problem with the connectivity between the client and server, e.g. the proxy under test is misconfigured or misbehaving, the polygraph server side can time out and shut down.

Delete actions on the master should tolerate the client or server reporting that they are unable to delete the run in question and still continue trying to clean up the run locally.

In the meantime, the workaround is to (in order, since some steps may unblock later steps):

  1. SSH onto the server, check for and kill any running polygraph processes, and clean out /run/plumage/server/.
  2. Repeat for the client and /run/plumage/client/. If there were polygraph processes to kill, be patient; this may unjam the process and start reporting results back to the master.
  3. SSH onto the master, look under /var/lib/plumage/configurations/N/runs/ (where N is in the URL of the configuration), and delete the directory matching the broken run number.
@LionsPhil LionsPhil added this to the v0.3: No more SSH milestone Jun 14, 2016
@LionsPhil LionsPhil added the bug label Jun 14, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant