Skip to content

7 MovedOrRenamed for Java-WebSocket#1545

Merged
darko-marinov merged 1 commit into
TestingResearchIllinois:mainfrom
Derrick2000:nov29
Dec 4, 2024
Merged

7 MovedOrRenamed for Java-WebSocket#1545
darko-marinov merged 1 commit into
TestingResearchIllinois:mainfrom
Derrick2000:nov29

Conversation

@Derrick2000
Copy link
Copy Markdown
Contributor

No description provided.

@darko-marinov
Copy link
Copy Markdown
Contributor

What do you get for git show --name-status 2ce6e6149d0ef44f4919e75326265c64c144a00f | grep Issue580Test? Can you check if the tests are still ID in the commit after the move/rename, or at the very least check if they're still ID in the latest commit?

@Derrick2000
Copy link
Copy Markdown
Contributor Author

Derrick2000 commented Nov 30, 2024

$ git show --name-status 2ce6e6149d0ef44f4919e75326265c64c144a00f | grep Issue580Test
M       src/test/java/org/java_websocket/issues/Issue580Test.java

In the latest commit:
Not ID flaky (Nondex passed) at: /home/mingyu7/Java-WebSocket/nondex-1732930910.log
Maybe still NOD flaky (Idflakies failed) at: /home/mingyu7/Java-WebSocket/failing-test-output-1732931252546-13955426-17a6-48e3-bd72-2078e5ae7742

Detected incompatible test case (org.java_websocket.issues.Issue580Test.runCloseBlockingTestScenario)
Detected incompatible test case (org.java_websocket.issues.Issue580Test.runNoCloseBlockingTestScenario)

VM: 056

@darko-marinov
Copy link
Copy Markdown
Contributor

Sorry for asking about ID; I didn't notice that the tests were NOD. Can you run them many times in the latest commit, even without iDFlakies but ideally with iDFlakies, to see if they still get reported? At least you can run for i in {1..10}; do mvn test -pl MODULE; done.

@Derrick2000
Copy link
Copy Markdown
Contributor Author

Nothing gets reported for mvn test at /home/mingyu7/Java-WebSocket/run_tests-1732936175.log
But with iDFlakies, issues get reported as above (even without using iteration)

@darko-marinov
Copy link
Copy Markdown
Contributor

I don't know what Detected incompatible test case means. Can you debug some more to find out?

Is iDFlakies saying that it cannot run that test? Why does it differ from the others?

Or is that some failure message from the test itself? Did iDFlakies find some OD or NOD test?

@Derrick2000
Copy link
Copy Markdown
Contributor Author

I believe the Detected incompatible test case indicates that the test may have characteristics that prevent it from being reliably analyzed for flakiness.

In our case,

  1. the issue is no longer reported after I removed the @Parameterized usage in the class at /home/mingyu7/Java-WebSocket/failing-test-output-1732949389175-b0b23ff4-1c21-45d3-859a-ea51d27fc25f
  2. mvn test still does not report any issue at /home/mingyu7/Java-WebSocket/run_tests-1732950081.log

@Derrick2000
Copy link
Copy Markdown
Contributor Author

Derrick2000 commented Dec 1, 2024

Automation script to find the commit that renamed or moved the test method:

VM: 056
Script: /home/mingyu7/Java-WebSocket/find_renamed_or_removed_commit.sh

Example:
FILE_PATH="src/test/java/org/java_websocket/issues/Issue580Test.java" FUNCTION_NAME="runNoCloseBlockingTestScenario0"

image

@Derrick2000
Copy link
Copy Markdown
Contributor Author

Added a automation script at: PR

@darko-marinov
Copy link
Copy Markdown
Contributor

Accepting this for now. Someone else will review the script in the other PR.

@darko-marinov darko-marinov merged commit a0eb903 into TestingResearchIllinois:main Dec 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants