Skip to content

Lin Bytes negative test fails to trigger on MSVC bytecode #518

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jmid opened this issue Jan 15, 2025 · 2 comments · Fixed by #519 or #546
Closed

Lin Bytes negative test fails to trigger on MSVC bytecode #518

jmid opened this issue Jan 15, 2025 · 2 comments · Fixed by #519 or #546
Labels
test suite reliability Issue concerns tests that should behave more predictably

Comments

@jmid
Copy link
Collaborator

jmid commented Jan 15, 2025

#516 changed the Lin Bytes test, turning it into a negative test.

The merge to main of the PR then failed to find a counterexample within 1000 attempts on MSVC bytecode 5.3:
https://github.com/ocaml-multicore/multicoretests/actions/runs/12791598373/job/35660087124

random seed: 278988859
generated error fail pass / total     time test name

[ ]    0    0    0    0 / 1000     0.0s Lin Bytes test with Domain
[ ]    0    0    0    0 / 1000     0.0s Lin Bytes test with Domain (generating)
[ ]   75    0    0   75 / 1000    60.1s Lin Bytes test with Domain
[ ]  343    0    0  343 / 1000   121.0s Lin Bytes test with Domain
[ ]  926    0    0  926 / 1000   181.0s Lin Bytes test with Domain
[✗] 1000    0    0 1000 / 1000   188.4s Lin Bytes test with Domain

[ ]    0    0    0    0 /  250     0.0s Lin Bytes test with Thread
[ ]   69    0    0   69 /  250   243.1s Lin Bytes test with Thread
[✓]  250    0    0  250 /  250   300.4s Lin Bytes test with Thread

--- Failure --------------------------------------------------------------------

Test Lin Bytes test with Domain failed:

Negative test Lin Bytes test with Domain succeeded but was expected to fail
================================================================================
failure (1 tests failed, 0 tests errored, ran 2 tests)
File "src/bytes/dune", line 12, characters 7-16:
12 |  (name lin_tests)
            ^^^^^^^^^
(cd _build/default/src/bytes && ./lin_tests.exe --verbose)
Command exited with code 1.

We should just increase the count for a start, as it is not taking particularly long to execute - and doesn't cost anything for platforms finding a counterexample earlier 🤷

@jmid jmid added the test suite reliability Issue concerns tests that should behave more predictably label Jan 15, 2025
@jmid jmid closed this as completed in #519 Jan 16, 2025
@jmid jmid reopened this Jan 22, 2025
@jmid
Copy link
Collaborator Author

jmid commented Jan 22, 2025

Reopening this as this happened on the push to main in connection with the 0.7 release for MSVC bytecode trunk:
https://github.com/ocaml-multicore/multicoretests/actions/runs/12891458000/job/35943440732

random seed: 318271471
generated error fail pass / total     time test name

[ ]    0    0    0    0 / 5000     0.0s Lin Bytes test with Domain
[ ]    0    0    0    0 / 5000     0.0s Lin Bytes test with Domain (generating)
[ ]  609    0    0  609 / 5000    88.8s Lin Bytes test with Domain
[ ]  902    0    0  902 / 5000   157.6s Lin Bytes test with Domain
[ ] 1208    0    0 1208 / 5000   258.0s Lin Bytes test with Domain
[ ] 1735    0    0 1735 / 5000   903.8s Lin Bytes test with Domain
[ ] 2176    0    0 2176 / 5000  1013.6s Lin Bytes test with Domain
[ ] 3118    0    0 3118 / 5000  1077.0s Lin Bytes test with Domain
[ ] 3493    0    0 3493 / 5000  1138.1s Lin Bytes test with Domain
[ ] 4085    0    0 4085 / 5000  1199.2s Lin Bytes test with Domain
[ ] 4247    0    0 4247 / 5000  1373.2s Lin Bytes test with Domain
[✗] 5000    0    0 5000 / 5000  1422.3s Lin Bytes test with Domain

I wonder if there's something fundamental with (MSVC) bytecode or whether it can be solved, e.g., by adjusting weights a bit like src/dynarray/lin_tests.ml does... 🤔

@jmid
Copy link
Collaborator Author

jmid commented Feb 24, 2025

Saw this again on the scheduled 5.3 MSVC bytecode run:
https://github.com/ocaml-multicore/multicoretests/actions/runs/13489954915/job/37686516127

random seed: 478587156
generated error fail pass / total     time test name

[ ]    0    0    0    0 / 5000     0.0s Lin Bytes test with Domain
[ ]    0    0    0    0 / 5000     0.0s Lin Bytes test with Domain (generating)
[ ]  852    0    0  852 / 5000    60.0s Lin Bytes test with Domain
[ ] 1167    0    0 1167 / 5000   120.0s Lin Bytes test with Domain
[ ] 1547    0    0 1547 / 5000   180.0s Lin Bytes test with Domain
[ ] 2300    0    0 2300 / 5000   245.6s Lin Bytes test with Domain
[ ] 2949    0    0 2949 / 5000   329.9s Lin Bytes test with Domain
[ ] 3532    0    0 3532 / 5000   390.9s Lin Bytes test with Domain
[ ] 3967    0    0 3967 / 5000   460.7s Lin Bytes test with Domain
[ ] 4618    0    0 4618 / 5000   520.9s Lin Bytes test with Domain
[✗] 5000    0    0 5000 / 5000   557.5s Lin Bytes test with Domain

The common trait here seems to be MSVC and bytecode... 🤔

@jmid jmid changed the title Lin Bytes negative test fails to trigger Lin Bytes negative test fails to trigger on MSVC bytecode Feb 24, 2025
@jmid jmid linked a pull request Mar 28, 2025 that will close this issue
@jmid jmid closed this as completed in #546 Mar 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
test suite reliability Issue concerns tests that should behave more predictably
Projects
None yet
1 participant