Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed a potential bugs that may lead to positional misalignment between pred and gt on different device. #111

Merged
merged 2 commits into from
Jan 16, 2024

Conversation

Siyuexi
Copy link
Contributor

@Siyuexi Siyuexi commented Jan 16, 2024

In the existing version, when DivLog performs an evaluation on the current results, it reorders "gt" according to the DPP algorithm. However, the DPP algorithm is implemented using the math library. We discovered that the computation results for DPP vary across different versions of Python, which subsequently leads to changes in the order of "gt".

To be more specific, if the parsed results are generated on device A, and the comparison with "gt" for evaluation is also conducted on device A, then the evaluation results are accurate. That's because both processes are operated on device A, thus ensuring the same DPP rearrangement order. However, if the results are generated on device A, but compared with "gt" on device B, the orders of results and "gt" may be misaligned, leading to incorrect evaluation results. This is due to the variations in DPP outcomes between the two devices, resulting in inconsistencies between the orders of samples in results and "gt".

To resolve this issue, the current update abandons ordered one-to-one evaluation according to the post-DPP mapping. Instead, it assesses based on whether the corresponding result and "gt" for each original log message are consistent. The current evaluation method has been corrected.

Additionally, in this update, we've added a notice of GPT-3's deprecation to the README.md.

…may cause positional misalignment between existing prediction and groundtruth on different platform with different python version.
@PinjiaHe PinjiaHe merged commit 7e6f6cd into logpai:main Jan 16, 2024
6 checks passed
@zhujiem
Copy link
Member

zhujiem commented Jan 16, 2024

  1. 后面PR尽量只保留一个commit,采用git commit -a --amend进行修改。@Siyuexi
  2. 如果是多个commit提交一个PR,可以选Squash这个选型进行合并 @PinjiaHe
  3. Readme中说明测试的python具体版本,因为tests中没有进行python版本覆盖测试,后期可能因为python版本导致error

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants