Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make occasionally failing ct testcases more robust #102

Open
gomoripeti opened this issue Oct 24, 2017 · 0 comments
Open

Make occasionally failing ct testcases more robust #102

gomoripeti opened this issue Oct 24, 2017 · 0 comments

Comments

@gomoripeti
Copy link
Collaborator

gomoripeti commented Oct 24, 2017

Some of the ct testcases fail from time to time but not deterministically

This failure is quite common and hopeless to fix because of how timers work in Erlang. (A call to timer:sleep(20) is guaranteed to last at least 20 ms but there is no upper limit)

%%% xprof_tracing_SUITE ==> long_call: FAILED
%%% xprof_tracing_SUITE ==> 
Failure/Error: ?assertMatch({ true , _ }, { Min < 22 * 1000 , Min })
  expected: = { true , _ }
       got: {false,24480}      line: 396

Another example when samples are not yet available. (This happens quite often in long_call testcase)

%%% xprof_http_e2e_SUITE ==> capture_data_when_traced_test: FAILED
%%% xprof_http_e2e_SUITE ==>
Failure/Error: ?assertEqual(1, length ( proplists : get_value ( << "items" >> , Data ) ))
expected: 1
got: 0 line: 210

To make snapshots more deterministic (without waiting 1-2 seconds) is to add a function to xprof_core_trace_handler to trigger taking one synchronously (and callerlang:trace_delivered before). But as the key of the snapshots is the timestamp in seconds, there can only be one snapshot per second (and some testcases require two).

Or captured data is not yet available:

%%% xprof_http_e2e_SUITE ==> capture_data_with_formatted_exception_test: FAILED
%%% xprof_http_e2e_SUITE ==>
Failure/Error: ?assertMatch([ << "** exception error: no match of right hand side value ok" >> ], [ proplists : get_value ( << "res" >> , Item ) || Item <- proplists : get_value ( << "items" >> , Data ) ])
expected: = [ << "** exception error: no match of right hand side value ok" >> ]
got: [] line: 244
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant