execute_command_in_target times out after 1 hour #8
Description
Keyword 'execute_command_in_target' did not obey the given timeout=600 argument but failed after ~1 hour 30 seconds.
This error happens occasionally, unfortunately some of the logs were silenced during the test run below:
KEYWORD RunnerExecutor . Execute Command In Target With Ssh To Node ${collect_command}, ${node}
Start / End / Elapsed: 20190226 16:33:08.564 / 20190226 17:33:42.585 / 01:00:34.021
16:33:08.564 TRACE Arguments: [ "<chain of mkdir, rm, cp, journalctl operations here>" | b'xyz.local' ]
16:33:08.564 DEBUG execute_command_in_target(command='ssh -4q -o BatchMode=no -o PasswordAuthentication=no -o PreferredAuthentications=publickey -o CheckHostIP=no 10.9.8.7 "<chain of mkdir, rm, cp, journalctl operations here>"', target='1.2.3.4')
17:33:39.573 DEBUG Popping shell 'MsgPythonShell'
17:33:42.581 INFO Failed to close terminal: TIMEOUT, console log:
17:33:42.584 TRACE Return: None
17:33:42.584 FAIL RunnerTerminalSessionBroken: TIMEOUT, console log:
17:33:42.584 DEBUG Traceback (most recent call last):
<---RunnerExecutor is calling execute_command_in_target with timeout=600--->
File "<removed>/virtualenv/lib/python2.7/site-packages/crl/interactivesessions/remoterunner.py", line 396, in execute_command_in_target
progress_log=progress_log))
File "<removed>/virtualenv/lib/python2.7/site-packages/crl/interactivesessions/_runnerintarget.py", line 34, in run
timeout=timeout).run()
File "<removed>/virtualenv/lib/python2.7/site-packages/crl/interactivesessions/_process.py", line 84, in run
self._communicate = self.pro.communicate
File "<removed>/virtualenv/lib/python2.7/site-packages/crl/interactivesessions/remoteproxies.py", line 16, in inner_function
return f(proxy, *args, **kwargs)
File "<removed>/virtualenv/lib/python2.7/site-packages/crl/interactivesessions/remoteproxies.py", line 362, in __getattr__
lambda response: self.__get_recursive_proxy_for_remotename(
File "<removed>/virtualenv/lib/python2.7/site-packages/crl/interactivesessions/runnerexceptions.py", line 73, in responsehandler
return response_wrap(function())
File "<removed>/virtualenv/lib/python2.7/site-packages/crl/interactivesessions/remoteproxies.py", line 360, in <lambda>
remotename),
File "<removed>/virtualenv/lib/python2.7/site-packages/crl/interactivesessions/runnerterminal.py", line 337, in run_and_return_handled_python
handled_types=self._get_python_arg(self.HANDLED_TYPES)).run()
File "<removed>/virtualenv/lib/python2.7/site-packages/crl/interactivesessions/runnerterminal.py", line 74, in run
self.cmd, timeout=self.run_timeout),
File "<removed>/virtualenv/lib/python2.7/site-packages/crl/interactivesessions/runnerterminal.py", line 221, in run
self._raise_session_broken_exception(e)
File "<removed>/virtualenv/lib/python2.7/site-packages/crl/interactivesessions/runnerterminal.py", line 237, in _raise_session_broken_exception
raise RunnerTerminalSessionBroken(exception)
Version 1.0.2 was used during this run.