Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes Probe logging for all iterations #676

Merged
merged 10 commits into from
Jan 11, 2024

Conversation

Nageshbansal
Copy link
Contributor

@Nageshbansal Nageshbansal commented Nov 27, 2023

What this PR does / why we need it:

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #678

Special notes for your reviewer:

Checklist:

  • Fixes #
  • PR messages has document related information
  • Labelled this PR & related issue with breaking-changes tag
  • PR messages has breaking changes related information
  • Labelled this PR & related issue with requires-upgrade tag
  • Commit has unit tests
  • PR messages has upgrade related information
  • Commit has integration tests
  • E2E run Required for the changes

@Nageshbansal
Copy link
Contributor Author

Nageshbansal commented Dec 13, 2023

httpProbe logging :
image

Signed-off-by: nagesh bansal <[email protected]>
pkg/probe/httpprobe.go Outdated Show resolved Hide resolved
pkg/probe/httpprobe.go Outdated Show resolved Hide resolved
@Nageshbansal
Copy link
Contributor Author

Experiment logs for HTTP Continuous Probe:

time="2023-12-16T21:16:01Z" level=info msg="Experiment Name: pod-delete"
time="2023-12-16T21:16:01Z" level=info msg="[PreReq]: Getting the ENV for the pod-delete experiment"
time="2023-12-16T21:16:03Z" level=info msg="[PreReq]: Updating the chaos result of pod-delete experiment (SOT)"
time="2023-12-16T21:16:07Z" level=info msg="The application information is as follows" Targets="[{namespace: litmus, kind: deployment, labels: [app=nginx]}]" Chaos Duration=30
time="2023-12-16T21:16:10Z" level=info msg="[Probe]: The http probe information is as follows" Run Properties="{10s 5s 1 0 2s   false}" Mode=Continuous Phase=PreChaos Name=http-probe URL="https://google.com"
time="2023-12-16T21:16:10Z" level=info msg="[Probe]: HTTP get method informations" ResponseCode=200 ResponseTimeout=10s Name=http-probe URL="https://google.com" Criteria="=="
time="2023-12-16T21:16:10Z" level=info msg="[Info]: The chaos tunables are:" PodsAffectedPerc=0 Sequence=parallel
time="2023-12-16T21:16:12Z" level=info msg="[Chaos]:Number of pods targeted: 1"
time="2023-12-16T21:16:12Z" level=info msg="Target pods list for chaos, [nginx-89bb6c4c-r82q8]"
time="2023-12-16T21:16:12Z" level=info msg="[Info]: Killing the following pods" PodName=nginx-89bb6c4c-r82q8
time="2023-12-16T21:16:12Z" level=info msg="[Wait]: Wait for the chaos interval 10s"
time="2023-12-16T21:16:12Z" level=info msg="[Probe]: {Actual value: 200}, {Expected value: 200}, {Operator: ==}"
time="2023-12-16T21:16:19Z" level=info msg="[Probe]: HTTP get method informations" Criteria="==" ResponseCode=200 ResponseTimeout=10s Name=http-probe URL="https://google.com"
time="2023-12-16T21:16:19Z" level=info msg="[Probe]: {Actual value: 200}, {Expected value: 200}, {Operator: ==}"
time="2023-12-16T21:16:22Z" level=info msg="[Status]: Verification for the recreation of application pod"
time="2023-12-16T21:16:22Z" level=info msg="[Status]: Checking whether application containers are in ready state"
time="2023-12-16T21:16:22Z" level=info msg="[Status]: The Container status are as follows" Pod=nginx-89bb6c4c-ft8l7 Readiness=true container=nginx
time="2023-12-16T21:16:24Z" level=info msg="[Status]: Checking whether application pods are in running state"
time="2023-12-16T21:16:24Z" level=info msg="[Status]: The status of Pods are as follows" Status=Running Pod=nginx-89bb6c4c-ft8l7
time="2023-12-16T21:16:26Z" level=info msg="[Chaos]:Number of pods targeted: 1"
time="2023-12-16T21:16:26Z" level=info msg="Target pods list for chaos, [nginx-89bb6c4c-ft8l7]"
time="2023-12-16T21:16:26Z" level=info msg="[Info]: Killing the following pods" PodName=nginx-89bb6c4c-ft8l7
time="2023-12-16T21:16:26Z" level=info msg="[Wait]: Wait for the chaos interval 10s"
time="2023-12-16T21:16:26Z" level=info msg="[Probe]: HTTP get method informations" ResponseCode=200 ResponseTimeout=10s Name=http-probe URL="https://google.com" Criteria="=="
time="2023-12-16T21:16:27Z" level=info msg="[Probe]: {Actual value: 200}, {Expected value: 200}, {Operator: ==}"
time="2023-12-16T21:16:34Z" level=info msg="[Probe]: HTTP get method informations" Name=http-probe URL="https://google.com" Criteria="==" ResponseCode=200 ResponseTimeout=10s
time="2023-12-16T21:16:35Z" level=info msg="[Probe]: {Actual value: 200}, {Expected value: 200}, {Operator: ==}"
time="2023-12-16T21:16:36Z" level=info msg="[Status]: Verification for the recreation of application pod"
time="2023-12-16T21:16:36Z" level=info msg="[Status]: Checking whether application containers are in ready state"
time="2023-12-16T21:16:36Z" level=info msg="[Status]: The Container status are as follows" container=nginx Pod=nginx-89bb6c4c-rqpkv Readiness=true
time="2023-12-16T21:16:38Z" level=info msg="[Status]: Checking whether application pods are in running state"
time="2023-12-16T21:16:38Z" level=info msg="[Status]: The status of Pods are as follows" Pod=nginx-89bb6c4c-rqpkv Status=Running
time="2023-12-16T21:16:40Z" level=info msg="[Chaos]:Number of pods targeted: 1"
time="2023-12-16T21:16:40Z" level=info msg="Target pods list for chaos, [nginx-89bb6c4c-rqpkv]"
time="2023-12-16T21:16:40Z" level=info msg="[Info]: Killing the following pods" PodName=nginx-89bb6c4c-rqpkv
time="2023-12-16T21:16:40Z" level=info msg="[Wait]: Wait for the chaos interval 10s"
time="2023-12-16T21:16:42Z" level=info msg="[Probe]: HTTP get method informations" Criteria="==" ResponseCode=200 ResponseTimeout=10s Name=http-probe URL="https://google.com"
time="2023-12-16T21:16:42Z" level=info msg="[Probe]: {Actual value: 200}, {Expected value: 200}, {Operator: ==}"
time="2023-12-16T21:16:49Z" level=info msg="[Probe]: HTTP get method informations" ResponseTimeout=10s Name=http-probe URL="https://google.com" Criteria="==" ResponseCode=200
time="2023-12-16T21:16:50Z" level=info msg="[Status]: Verification for the recreation of application pod"
time="2023-12-16T21:16:50Z" level=info msg="[Status]: Checking whether application containers are in ready state"
time="2023-12-16T21:16:50Z" level=info msg="[Status]: The Container status are as follows" container=nginx Pod=nginx-89bb6c4c-xfckq Readiness=true
time="2023-12-16T21:16:50Z" level=info msg="[Probe]: {Actual value: 200}, {Expected value: 200}, {Operator: ==}"
time="2023-12-16T21:16:52Z" level=info msg="[Status]: Checking whether application pods are in running state"
time="2023-12-16T21:16:52Z" level=info msg="[Status]: The status of Pods are as follows" Pod=nginx-89bb6c4c-xfckq Status=Running
time="2023-12-16T21:16:54Z" level=info msg="[Completion]: pod-delete chaos is done"
time="2023-12-16T21:16:54Z" level=info msg="[Confirmation]: pod-delete chaos has been injected successfully"
time="2023-12-16T21:16:56Z" level=info msg="[Probe]: Waiting for http-probe probe to finish or timeout"
time="2023-12-16T21:16:57Z" level=info msg="Chaos Execution completed. Stopping Probes"
time="2023-12-16T21:16:58Z" level=info msg="[Probe]: http-probe probe has been Passed 😄 " ProbeType=httpProbe ProbeInstance=PostChaos ProbeStatus=Passed ProbeName=http-probe
time="2023-12-16T21:16:58Z" level=info msg="[The End]: Updating the chaos result of pod-delete experiment (EOT)"

@Nageshbansal
Copy link
Contributor Author

PromeProbe:

time="2023-12-19T20:06:32Z" level=info msg="Experiment Name: pod-delete"
time="2023-12-19T20:06:32Z" level=info msg="[PreReq]: Getting the ENV for the pod-delete experiment"
time="2023-12-19T20:06:34Z" level=info msg="[PreReq]: Updating the chaos result of pod-delete experiment (SOT)"
time="2023-12-19T20:06:38Z" level=info msg="The application information is as follows" Chaos Duration=10 Targets="[{namespace: litmus, kind: deployment, labels: [app=nginx]}]"
time="2023-12-19T20:06:40Z" level=info msg="[Probe]: The prometheus probe information is as follows" Comparator="{ > 0}" Run Properties="{10s 5s 1 0 2s   false}" Mode=Continuous Phase=PreChaos Name=prom-probe Query="sum by (namespace) (kube_pod_container_status_running{namespace=\\\"litmus\\\"})" Endpoint="http://192.168.49.2:30556"
time="2023-12-19T20:06:40Z" level=info msg="[Info]: The chaos tunables are:" PodsAffectedPerc=0 Sequence=parallel
time="2023-12-19T20:06:40Z" level=info msg="[Probe]: {Actual value: 3}, {Expected value: 0}, {Operator: >}"
time="2023-12-19T20:06:42Z" level=info msg="[Chaos]:Number of pods targeted: 1"
time="2023-12-19T20:06:42Z" level=info msg="Target pods list for chaos, [nginx-89bb6c4c-krnfj]"
time="2023-12-19T20:06:42Z" level=info msg="[Info]: Killing the following pods" PodName=nginx-89bb6c4c-krnfj
time="2023-12-19T20:06:42Z" level=info msg="[Wait]: Wait for the chaos interval 10s"
time="2023-12-19T20:06:42Z" level=info msg="[Probe]: {Actual value: 3}, {Expected value: 0}, {Operator: >}"
time="2023-12-19T20:06:44Z" level=info msg="[Probe]: {Actual value: 3}, {Expected value: 0}, {Operator: >}"
time="2023-12-19T20:06:46Z" level=info msg="[Probe]: {Actual value: 3}, {Expected value: 0}, {Operator: >}"
time="2023-12-19T20:06:48Z" level=info msg="[Probe]: {Actual value: 3}, {Expected value: 0}, {Operator: >}"
time="2023-12-19T20:06:50Z" level=info msg="[Probe]: {Actual value: 3}, {Expected value: 0}, {Operator: >}"
time="2023-12-19T20:06:52Z" level=info msg="[Status]: Verification for the recreation of application pod"
time="2023-12-19T20:06:52Z" level=info msg="[Status]: Checking whether application containers are in ready state"
time="2023-12-19T20:06:52Z" level=info msg="[Status]: The Container status are as follows" Pod=nginx-89bb6c4c-f2zcf Readiness=true container=nginx
time="2023-12-19T20:06:52Z" level=info msg="[Probe]: {Actual value: 3}, {Expected value: 0}, {Operator: >}"
time="2023-12-19T20:06:54Z" level=info msg="[Status]: Checking whether application pods are in running state"
time="2023-12-19T20:06:54Z" level=info msg="[Status]: The status of Pods are as follows" Pod=nginx-89bb6c4c-f2zcf Status=Running
time="2023-12-19T20:06:54Z" level=info msg="[Probe]: {Actual value: 3}, {Expected value: 0}, {Operator: >}"
time="2023-12-19T20:06:56Z" level=info msg="[Completion]: pod-delete chaos is done"
time="2023-12-19T20:06:56Z" level=info msg="[Confirmation]: pod-delete chaos has been injected successfully"
time="2023-12-19T20:06:56Z" level=info msg="[Probe]: {Actual value: 4}, {Expected value: 0}, {Operator: >}"
time="2023-12-19T20:06:58Z" level=info msg="[Probe]: Waiting for prom-probe probe to finish or timeout"
time="2023-12-19T20:06:58Z" level=info msg="Chaos Execution completed. Stopping Probes"
time="2023-12-19T20:07:00Z" level=info msg="[Probe]: prom-probe probe has been Passed 😄 " ProbeName=prom-probe ProbeType=promProbe ProbeInstance=PostChaos ProbeStatus=Passed
time="2023-12-19T20:07:00Z" level=info msg="[The End]: Updating the chaos result of pod-delete experiment (EOT)"

@Nageshbansal
Copy link
Contributor Author

cmdProbe ( source mode):

ime="2023-12-19T20:08:41Z" level=info msg="Experiment Name: pod-delete"
time="2023-12-19T20:08:41Z" level=info msg="[PreReq]: Getting the ENV for the pod-delete experiment"
time="2023-12-19T20:08:43Z" level=info msg="[PreReq]: Updating the chaos result of pod-delete experiment (SOT)"
time="2023-12-19T20:08:47Z" level=info msg="The application information is as follows" Targets="[{namespace: litmus, kind: deployment, labels: [app=nginx]}]" Chaos Duration=10
time="2023-12-19T20:08:49Z" level=info msg="[Probe]: The cmd probe information is as follows" Name=cmd-probe-source Command="echo 'hello'" Comparator="{string contains hello}" Source="&{nginx false false [] [] map[] map[] []  false map[] [] [] []}" Run Properties="{10s 5s 1 0 2s   false}" Mode=Continuous Phase=PreChaos
time="2023-12-19T20:08:49Z" level=info msg="[Status]: Checking the status of the probe pod"
time="2023-12-19T20:08:49Z" level=info msg="[Status]: Checking whether application containers are in ready state"
time="2023-12-19T20:08:51Z" level=info msg="[Status]: Checking whether application pods are in running state"
time="2023-12-19T20:08:53Z" level=info msg="[Status]: The status of Pods are as follows" Pod=pod-delete-probe-n3uvnh Status=Running
time="2023-12-19T20:08:55Z" level=info msg="[Info]: The chaos tunables are:" Sequence=parallel PodsAffectedPerc=0
time="2023-12-19T20:08:55Z" level=info msg="[Probe]: {Actual value: hello}, {Expected value: hello}, {Operator: contains}"
time="2023-12-19T20:08:57Z" level=info msg="[Chaos]:Number of pods targeted: 1"
time="2023-12-19T20:08:57Z" level=info msg="Target pods list for chaos, [nginx-89bb6c4c-f2zcf]"
time="2023-12-19T20:08:57Z" level=info msg="[Info]: Killing the following pods" PodName=nginx-89bb6c4c-f2zcf
time="2023-12-19T20:08:57Z" level=info msg="[Wait]: Wait for the chaos interval 10s"
time="2023-12-19T20:08:57Z" level=info msg="[Probe]: {Actual value: hello}, {Expected value: hello}, {Operator: contains}"
time="2023-12-19T20:08:59Z" level=info msg="[Probe]: {Actual value: hello}, {Expected value: hello}, {Operator: contains}"
time="2023-12-19T20:09:01Z" level=info msg="[Probe]: {Actual value: hello}, {Expected value: hello}, {Operator: contains}"
time="2023-12-19T20:09:04Z" level=info msg="[Probe]: {Actual value: hello}, {Expected value: hello}, {Operator: contains}"
time="2023-12-19T20:09:06Z" level=info msg="[Probe]: {Actual value: hello}, {Expected value: hello}, {Operator: contains}"
time="2023-12-19T20:09:07Z" level=info msg="[Status]: Verification for the recreation of application pod"
time="2023-12-19T20:09:07Z" level=info msg="[Status]: Checking whether application containers are in ready state"
time="2023-12-19T20:09:07Z" level=info msg="[Status]: The Container status are as follows" Readiness=true container=nginx Pod=nginx-89bb6c4c-cj68g
time="2023-12-19T20:09:08Z" level=info msg="[Probe]: {Actual value: hello}, {Expected value: hello}, {Operator: contains}"
time="2023-12-19T20:09:09Z" level=info msg="[Status]: Checking whether application pods are in running state"
time="2023-12-19T20:09:09Z" level=info msg="[Status]: The status of Pods are as follows" Pod=nginx-89bb6c4c-cj68g Status=Running
time="2023-12-19T20:09:10Z" level=info msg="[Probe]: {Actual value: hello}, {Expected value: hello}, {Operator: contains}"
time="2023-12-19T20:09:11Z" level=info msg="[Completion]: pod-delete chaos is done"
time="2023-12-19T20:09:11Z" level=info msg="[Confirmation]: pod-delete chaos has been injected successfully"
time="2023-12-19T20:09:12Z" level=info msg="[Probe]: {Actual value: hello}, {Expected value: hello}, {Operator: contains}"
time="2023-12-19T20:09:13Z" level=info msg="[Probe]: Waiting for cmd-probe-source probe to finish or timeout"
time="2023-12-19T20:09:14Z" level=info msg="[Probe]: Chaos Execution completed. Stopping Probes"
time="2023-12-19T20:09:15Z" level=info msg="[Probe]: cmd-probe-source probe has been Passed 😄 " ProbeName=cmd-probe-source ProbeType=cmdProbe ProbeInstance=PostChaos ProbeStatus=Passed
time="2023-12-19T20:09:17Z" level=info msg="[The End]: Updating the chaos result of pod-delete experiment (EOT)"

cmdProbe (inline-mode)

time="2023-12-19T20:10:41Z" level=info msg="Experiment Name: pod-delete"
time="2023-12-19T20:10:41Z" level=info msg="[PreReq]: Getting the ENV for the pod-delete experiment"
time="2023-12-19T20:10:43Z" level=info msg="[PreReq]: Updating the chaos result of pod-delete experiment (SOT)"
time="2023-12-19T20:10:47Z" level=info msg="The application information is as follows" Targets="[{namespace: litmus, kind: deployment, labels: [app=nginx]}]" Chaos Duration=10
time="2023-12-19T20:10:49Z" level=info msg="[Probe]: The cmd probe information is as follows" Run Properties="{10s 5s 1 0 2s   false}" Mode=Continuous Phase=PreChaos Name=cmd-probe-inline Command="echo 'hello'" Comparator="{string contains hello}" Source="<nil>"
time="2023-12-19T20:10:49Z" level=info msg="[Info]: The chaos tunables are:" PodsAffectedPerc=0 Sequence=parallel
time="2023-12-19T20:10:49Z" level=info msg="[Probe]: {Actual value: hello}, {Expected value: hello}, {Operator: contains}"
time="2023-12-19T20:10:51Z" level=info msg="[Probe]: {Actual value: hello}, {Expected value: hello}, {Operator: contains}"
time="2023-12-19T20:10:51Z" level=info msg="[Chaos]:Number of pods targeted: 1"
time="2023-12-19T20:10:51Z" level=info msg="Target pods list for chaos, [nginx-89bb6c4c-cj68g]"
time="2023-12-19T20:10:51Z" level=info msg="[Info]: Killing the following pods" PodName=nginx-89bb6c4c-cj68g
time="2023-12-19T20:10:51Z" level=info msg="[Wait]: Wait for the chaos interval 10s"
time="2023-12-19T20:10:53Z" level=info msg="[Probe]: {Actual value: hello}, {Expected value: hello}, {Operator: contains}"
time="2023-12-19T20:10:55Z" level=info msg="[Probe]: {Actual value: hello}, {Expected value: hello}, {Operator: contains}"
time="2023-12-19T20:10:57Z" level=info msg="[Probe]: {Actual value: hello}, {Expected value: hello}, {Operator: contains}"
time="2023-12-19T20:10:59Z" level=info msg="[Probe]: {Actual value: hello}, {Expected value: hello}, {Operator: contains}"
time="2023-12-19T20:11:01Z" level=info msg="[Probe]: {Actual value: hello}, {Expected value: hello}, {Operator: contains}"
time="2023-12-19T20:11:01Z" level=info msg="[Status]: Verification for the recreation of application pod"
time="2023-12-19T20:11:01Z" level=info msg="[Status]: Checking whether application containers are in ready state"
time="2023-12-19T20:11:01Z" level=info msg="[Status]: The Container status are as follows" container=nginx Pod=nginx-89bb6c4c-dkxp6 Readiness=true
time="2023-12-19T20:11:03Z" level=info msg="[Probe]: {Actual value: hello}, {Expected value: hello}, {Operator: contains}"
time="2023-12-19T20:11:03Z" level=info msg="[Status]: Checking whether application pods are in running state"
time="2023-12-19T20:11:03Z" level=info msg="[Status]: The status of Pods are as follows" Pod=nginx-89bb6c4c-dkxp6 Status=Running
time="2023-12-19T20:11:05Z" level=info msg="[Probe]: {Actual value: hello}, {Expected value: hello}, {Operator: contains}"
time="2023-12-19T20:11:05Z" level=info msg="[Completion]: pod-delete chaos is done"
time="2023-12-19T20:11:05Z" level=info msg="[Confirmation]: pod-delete chaos has been injected successfully"
time="2023-12-19T20:11:07Z" level=info msg="[Probe]: {Actual value: hello}, {Expected value: hello}, {Operator: contains}"
time="2023-12-19T20:11:07Z" level=info msg="[Probe]: Waiting for cmd-probe-inline probe to finish or timeout"
time="2023-12-19T20:11:09Z" level=info msg="Chaos Execution completed. Stopping Probes"
time="2023-12-19T20:11:09Z" level=info msg="[Probe]: cmd-probe-inline probe has been Passed 😄 " ProbeName=cmd-probe-inline ProbeType=cmdProbe ProbeInstance=PostChaos ProbeStatus=Passed
time="2023-12-19T20:11:09Z" level=info msg="[The End]: Updating the chaos result of pod-delete experiment (EOT)"

pkg/probe/cmdprobe.go Outdated Show resolved Hide resolved
pkg/probe/probe.go Outdated Show resolved Hide resolved
pkg/probe/cmdprobe.go Outdated Show resolved Hide resolved
@neelanjan00
Copy link
Member

@Nageshbansal let's refrain from adding multiple features in the same PR, it becomes difficult to track, manage, or potentially revert the PRs later. This is regarding the verbosity changes, ideally, we should create another PR.

@Nageshbansal
Copy link
Contributor Author

Reverting the changes for adding the verbosity.

@Nageshbansal
Copy link
Contributor Author

@neelanjan00 done with changes. PTAL :)

Signed-off-by: nagesh bansal <[email protected]>
Signed-off-by: nagesh bansal <[email protected]>
@neelanjan00
Copy link
Member

Reverting the changes for adding the verbosity.

Hi Nagesh, have you reverted the verbosity changes yet? I am still noticing them in the code changes.

@Nageshbansal
Copy link
Contributor Author

hey, I had a discussion with @ispeakc0de, and it was decided that the verbosity should be part of this PR only, as it is part of this change only. also a issue #678 has been created to track this PR. PTAL

Copy link
Member

@neelanjan00 neelanjan00 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Signed-off-by: nagesh bansal <[email protected]>
@ispeakc0de ispeakc0de merged commit 6992748 into litmuschaos:master Jan 11, 2024
4 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Log all iterations of the probe for onchaos and continuous modes configured via verbosity flag
3 participants