Skip to content

Conversation

@DemiMarie
Copy link
Contributor

The qrexec tests are broken and fail intermittently. This patch makes the bug obvious: by adding a sleep, it causes the test to fail right away, reliably.

The qrexec tests are broken and fail intermittently.  This patch makes
the bug obvious: by adding a sleep, it causes the test to fail right
away, reliably.
@codecov
Copy link

codecov bot commented Feb 22, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.72%. Comparing base (5de215d) to head (87f187d).
Report is 51 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #661      +/-   ##
==========================================
- Coverage   69.79%   69.72%   -0.07%     
==========================================
  Files          58       58              
  Lines       12546    12546              
==========================================
- Hits         8756     8748       -8     
- Misses       3790     3798       +8     
Flag Coverage Δ
unittests 69.72% <ø> (-0.07%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@qubesos-bot
Copy link

qubesos-bot commented Feb 23, 2025

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025041215-4.3&flavor=pull-requests

Test run included the following:

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025031804-4.3&flavor=update

  • system_tests_qwt_win11@hw13

    • windows_install: wait_serial (wait serial expected)
      # wait_serial expected: qr/dcWzE-\d+-/...

    • windows_install: Failed (test died + timed out)
      # Test died: command 'script -e -c 'bash -x /usr/bin/qvm-create-win...

  • system_tests_suspend@hw1

    • suspend: unnamed test (unknown)
    • suspend: Failed (test died)
      # Test died: no candidate needle with tag(s) 'SUSPEND-FAILED' match...
  • system_tests_whonix

    • whonixcheck: unnamed test (unknown)

    • whonixcheck: wait_serial (wait serial expected)
      # wait_serial expected: qr/TBvsr-\d+-/...

    • whonixcheck: Failed (test died + timed out)
      # Test died: command 'qvm-run -ap whonix-gateway-17 'LC_ALL=C whon...

  • system_tests_gui_tools

    • qui_widgets_update: unnamed test (unknown)
    • qui_widgets_update: Failed (test died)
      # Test died: no candidate needle with tag(s) 'qui-updates-widget-op...
  • system_tests_suspend

    • suspend: wait_serial (wait serial expected)
      # wait_serial expected: qr/wG5VX-\d+-/...
  • system_tests_extra

    • TC_00_QVCTest_whonix-workstation-17: test_010_screenshare (failure + cleanup)
      ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 0 == 0
  • system_tests_qrexec

  • system_tests_kde_gui_interactive

    • gui_keyboard_layout: wait_serial (wait serial expected)
      # wait_serial expected: "echo -e '[Layout]\nLayoutList=us,de' | sud...

    • gui_keyboard_layout: Failed (test died)
      # Test died: command 'test "$(cd ~user;ls e1*)" = "$(qvm-run -p wor...

  • system_tests_pvgrub_salt_storage

    • TC_10_VMSalt_fedora-41-xfce: test_003_update (failure)
      AssertionError: Salt command '['qubesctl', '--skip-dom0', '--show-o...
  • system_tests_audio

  • system_tests_guivm_vnc_gui_interactive

    • guivm_manager: unnamed test (unknown)
    • guivm_manager: Failed (test died)
      # Test died: no candidate needle with tag(s) 'menu-qubes-tools-subm...
  • system_tests_qwt_win10@hw13

    • windows_install: wait_serial (wait serial expected)
      # wait_serial expected: qr/LDWI1-\d+-/...

    • windows_install: Failed (test died + timed out)
      # Test died: command 'script -e -c 'bash -x /usr/bin/qvm-create-win...

  • system_tests_qwt_win10_seamless@hw13

    • windows_clipboard_and_filecopy: unnamed test (unknown)
    • windows_clipboard_and_filecopy: Failed (test died)
      # Test died: no candidate needle with tag(s) 'windows-Explorer-empt...

Failed tests

37 failures
  • system_tests_whonix@hw7

    • whonixcheck: fail (unknown)
      Whonixcheck for sys-whonix failed...

    • whonixcheck: unnamed test (unknown)

  • system_tests_qwt_win11@hw13

    • windows_install: wait_serial (wait serial expected)
      # wait_serial expected: qr/dcWzE-\d+-/...

    • windows_install: Failed (test died + timed out)
      # Test died: command 'script -e -c 'bash -x /usr/bin/qvm-create-win...

  • system_tests_suspend@hw1

    • suspend: unnamed test (unknown)
    • suspend: Failed (test died)
      # Test died: no candidate needle with tag(s) 'SUSPEND-FAILED' match...
  • system_tests_whonix

    • whonixcheck: fail (unknown)
      Whonixcheck for anon-whonix failed...

    • whonixcheck: unnamed test (unknown)

    • whonixcheck: wait_serial (wait serial expected)
      # wait_serial expected: qr/TBvsr-\d+-/...

    • whonixcheck: Failed (test died + timed out)
      # Test died: command 'qvm-run -ap whonix-gateway-17 'LC_ALL=C whon...

  • system_tests_gui_tools

    • qui_widgets_update: unnamed test (unknown)
    • qui_widgets_update: Failed (test died)
      # Test died: no candidate needle with tag(s) 'qui-updates-widget-op...
  • system_tests_suspend

    • suspend: wait_serial (wait serial expected)
      # wait_serial expected: qr/wG5VX-\d+-/...

    • suspend: Failed (test died + timed out)
      # Test died: command 'qvm-run -ap sys-whonix 'LC_ALL=C whonixcheck ...

  • system_tests_extra

    • TC_00_QVCTest_whonix-workstation-17: test_010_screenshare (failure + cleanup)
      ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 0 == 0
  • system_tests_qrexec

  • system_tests_kde_gui_interactive

    • gui_keyboard_layout: wait_serial (wait serial expected)
      # wait_serial expected: "echo -e '[Layout]\nLayoutList=us,de' | sud...

    • gui_keyboard_layout: Failed (test died)
      # Test died: command 'test "$(cd ~user;ls e1*)" = "$(qvm-run -p wor...

  • system_tests_pvgrub_salt_storage

    • TC_10_VMSalt_fedora-41-xfce: test_003_update (failure)
      AssertionError: Salt command '['qubesctl', '--skip-dom0', '--show-o...
  • system_tests_audio

  • system_tests_guivm_vnc_gui_interactive

    • guivm_manager: unnamed test (unknown)
    • guivm_manager: Failed (test died)
      # Test died: no candidate needle with tag(s) 'menu-qubes-tools-subm...
  • system_tests_qwt_win10@hw13

    • windows_install: wait_serial (wait serial expected)
      # wait_serial expected: qr/LDWI1-\d+-/...

    • windows_install: Failed (test died + timed out)
      # Test died: command 'script -e -c 'bash -x /usr/bin/qvm-create-win...

  • system_tests_qwt_win10_seamless@hw13

    • windows_clipboard_and_filecopy: unnamed test (unknown)
    • windows_clipboard_and_filecopy: Failed (test died)
      # Test died: no candidate needle with tag(s) 'windows-Explorer-empt...

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/132953#dependencies

9 fixed
  • system_tests_whonix

  • system_tests_suspend

  • system_tests_basic_vm_qrexec_gui

  • system_tests_kde_gui_interactive

    • clipboard_and_web: unnamed test (unknown)

    • clipboard_and_web: Failed (test died)
      # Test died: no candidate needle with tag(s) 'qubes-website' matche...

    • clipboard_and_web: wait_serial (wait serial expected)
      # wait_serial expected: "lspci; echo 2E8vz-\$?-"...

  • system_tests_audio

  • system_tests_guivm_vnc_gui_interactive

    • gui_filecopy: unnamed test (unknown)
    • gui_filecopy: Failed (test died)
      # Test died: no candidate needle with tag(s) 'files-work' matched...

Unstable tests

Performance Tests

Performance degradation:

16 performance degradations
  • debian-12-xfce_exec: 8.37 :small_red_triangle_up: ( previous job: 7.12, degradation: 117.52%)
  • whonix-gateway-17_exec: 8.90 :small_red_triangle_up: ( previous job: 6.82, degradation: 130.50%)
  • whonix-gateway-17_socket-root: 8.50 :small_red_triangle_up: ( previous job: 7.65, degradation: 111.16%)
  • whonix-workstation-17_socket-root: 9.04 :small_red_triangle_up: ( previous job: 8.13, degradation: 111.15%)
  • dom0_root_seq1m_q8t1_read 3:read_bandwidth_kb: 116150.00 :small_red_triangle_up: ( previous job: 446963.00, degradation: 25.99%)
  • dom0_root_seq1m_q1t1_read 3:read_bandwidth_kb: 31030.00 :small_red_triangle_up: ( previous job: 294295.00, degradation: 10.54%)
  • dom0_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 24355.00 :small_red_triangle_up: ( previous job: 79803.00, degradation: 30.52%)
  • dom0_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 950.00 :small_red_triangle_up: ( previous job: 6149.00, degradation: 15.45%)
  • dom0_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 2198.00 :small_red_triangle_up: ( previous job: 4826.00, degradation: 45.54%)
  • dom0_varlibqubes_seq1m_q8t1_write 3:write_bandwidth_kb: 149908.00 :small_red_triangle_up: ( previous job: 250795.00, degradation: 59.77%)
  • dom0_varlibqubes_rnd4k_q32t1_write 3:write_bandwidth_kb: 5338.00 :small_red_triangle_up: ( previous job: 6479.00, degradation: 82.39%)
  • dom0_varlibqubes_rnd4k_q1t1_write 3:write_bandwidth_kb: 3556.00 :small_red_triangle_up: ( previous job: 4903.00, degradation: 72.53%)
  • fedora-41-xfce_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 2140.00 :small_red_triangle_up: ( previous job: 3599.00, degradation: 59.46%)
  • fedora-41-xfce_private_seq1m_q1t1_read 3:read_bandwidth_kb: 275000.00 :small_red_triangle_up: ( previous job: 334687.00, degradation: 82.17%)
  • fedora-41-xfce_volatile_rnd4k_q32t1_write 3:write_bandwidth_kb: 1888.00 :small_red_triangle_up: ( previous job: 5672.00, degradation: 33.29%)
  • fedora-41-xfce_volatile_rnd4k_q1t1_write 3:write_bandwidth_kb: 1478.00 :small_red_triangle_up: ( previous job: 1953.00, degradation: 75.68%)

Remaining performance tests:

56 tests
  • debian-12-xfce_exec-root: 28.79 :small_red_triangle_up: ( previous job: 28.65, degradation: 100.45%)
  • debian-12-xfce_socket: 8.23 🟢 ( previous job: 8.60, improvement: 95.65%)
  • debian-12-xfce_socket-root: 8.92 :small_red_triangle_up: ( previous job: 8.52, degradation: 104.63%)
  • debian-12-xfce_exec-data-simplex: 69.28 🟢 ( previous job: 71.62, improvement: 96.72%)
  • debian-12-xfce_exec-data-duplex: 68.10 🟢 ( previous job: 70.34, improvement: 96.81%)
  • debian-12-xfce_exec-data-duplex-root: 86.76 :small_red_triangle_up: ( previous job: 82.72, degradation: 104.88%)
  • debian-12-xfce_socket-data-duplex: 163.59 :small_red_triangle_up: ( previous job: 156.96, degradation: 104.23%)
  • fedora-41-xfce_exec: 9.33 :small_red_triangle_up: ( previous job: 9.27, degradation: 100.66%)
  • fedora-41-xfce_exec-root: 63.23 :small_red_triangle_up: ( previous job: 61.51, degradation: 102.80%)
  • fedora-41-xfce_socket: 8.33 🟢 ( previous job: 8.63, improvement: 96.50%)
  • fedora-41-xfce_socket-root: 9.00 :small_red_triangle_up: ( previous job: 8.71, degradation: 103.34%)
  • fedora-41-xfce_exec-data-simplex: 64.75 🟢 ( previous job: 75.53, improvement: 85.73%)
  • fedora-41-xfce_exec-data-duplex: 66.18 🟢 ( previous job: 71.56, improvement: 92.48%)
  • fedora-41-xfce_exec-data-duplex-root: 105.60 🟢 ( previous job: 109.13, improvement: 96.77%)
  • fedora-41-xfce_socket-data-duplex: 149.04 🟢 ( previous job: 150.61, improvement: 98.96%)
  • whonix-gateway-17_exec-root: 40.34 🟢 ( previous job: 40.43, improvement: 99.76%)
  • whonix-gateway-17_socket: 7.41 :small_red_triangle_up: ( previous job: 7.24, degradation: 102.40%)
  • whonix-gateway-17_exec-data-simplex: 70.65 🟢 ( previous job: 78.32, improvement: 90.22%)
  • whonix-gateway-17_exec-data-duplex: 75.15 🟢 ( previous job: 76.65, improvement: 98.04%)
  • whonix-gateway-17_exec-data-duplex-root: 92.29 :small_red_triangle_up: ( previous job: 88.52, degradation: 104.26%)
  • whonix-gateway-17_socket-data-duplex: 170.79 🟢 ( previous job: 171.76, improvement: 99.43%)
  • whonix-workstation-17_exec: 8.03 :small_red_triangle_up: ( previous job: 7.67, degradation: 104.72%)
  • whonix-workstation-17_exec-root: 62.43 :small_red_triangle_up: ( previous job: 58.26, degradation: 107.15%)
  • whonix-workstation-17_socket: 8.50 :small_red_triangle_up: ( previous job: 8.19, degradation: 103.77%)
  • whonix-workstation-17_exec-data-simplex: 76.51 :small_red_triangle_up: ( previous job: 74.99, degradation: 102.02%)
  • whonix-workstation-17_exec-data-duplex: 64.36 🟢 ( previous job: 72.71, improvement: 88.52%)
  • whonix-workstation-17_exec-data-duplex-root: 102.71 :small_red_triangle_up: ( previous job: 99.82, degradation: 102.89%)
  • whonix-workstation-17_socket-data-duplex: 165.93 🟢 ( previous job: 169.50, improvement: 97.89%)
  • dom0_root_seq1m_q8t1_write 3:write_bandwidth_kb: 168170.00 :green_circle: ( previous job: 129298.00, improvement: 130.06%)
  • dom0_root_seq1m_q1t1_write 3:write_bandwidth_kb: 104179.00 :green_circle: ( previous job: 95454.00, improvement: 109.14%)
  • dom0_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 11162.00 :green_circle: ( previous job: 10795.00, improvement: 103.40%)
  • dom0_varlibqubes_seq1m_q8t1_read 3:read_bandwidth_kb: 466033.00 :green_circle: ( previous job: 382273.00, improvement: 121.91%)
  • dom0_varlibqubes_seq1m_q1t1_read 3:read_bandwidth_kb: 438734.00 :green_circle: ( previous job: 437636.00, improvement: 100.25%)
  • dom0_varlibqubes_seq1m_q1t1_write 3:write_bandwidth_kb: 180273.00 :small_red_triangle_up: ( previous job: 184752.00, degradation: 97.58%)
  • dom0_varlibqubes_rnd4k_q32t1_read 3:read_bandwidth_kb: 102966.00 :green_circle: ( previous job: 62195.00, improvement: 165.55%)
  • dom0_varlibqubes_rnd4k_q1t1_read 3:read_bandwidth_kb: 7688.00 :green_circle: ( previous job: 7669.00, improvement: 100.25%)
  • fedora-41-xfce_root_seq1m_q8t1_read 3:read_bandwidth_kb: 367277.00 :small_red_triangle_up: ( previous job: 368309.00, degradation: 99.72%)
  • fedora-41-xfce_root_seq1m_q8t1_write 3:write_bandwidth_kb: 280143.00 :green_circle: ( previous job: 162081.00, improvement: 172.84%)
  • fedora-41-xfce_root_seq1m_q1t1_read 3:read_bandwidth_kb: 323434.00 :green_circle: ( previous job: 318716.00, improvement: 101.48%)
  • fedora-41-xfce_root_seq1m_q1t1_write 3:write_bandwidth_kb: 111370.00 :green_circle: ( previous job: 87940.00, improvement: 126.64%)
  • fedora-41-xfce_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 93009.00 :green_circle: ( previous job: 82694.00, improvement: 112.47%)
  • fedora-41-xfce_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 7792.00 :small_red_triangle_up: ( previous job: 8485.00, degradation: 91.83%)
  • fedora-41-xfce_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 906.00 :green_circle: ( previous job: 542.00, improvement: 167.16%)
  • fedora-41-xfce_private_seq1m_q8t1_read 3:read_bandwidth_kb: 393609.00 :green_circle: ( previous job: 373957.00, improvement: 105.26%)
  • fedora-41-xfce_private_seq1m_q8t1_write 3:write_bandwidth_kb: 174844.00 :green_circle: ( previous job: 170062.00, improvement: 102.81%)
  • fedora-41-xfce_private_seq1m_q1t1_write 3:write_bandwidth_kb: 89487.00 :green_circle: ( previous job: 61534.00, improvement: 145.43%)
  • fedora-41-xfce_private_rnd4k_q32t1_read 3:read_bandwidth_kb: 74301.00 :small_red_triangle_up: ( previous job: 80283.00, degradation: 92.55%)
  • fedora-41-xfce_private_rnd4k_q32t1_write 3:write_bandwidth_kb: 3220.00 :green_circle: ( previous job: 2215.00, improvement: 145.37%)
  • fedora-41-xfce_private_rnd4k_q1t1_read 3:read_bandwidth_kb: 8522.00 :green_circle: ( previous job: 7540.00, improvement: 113.02%)
  • fedora-41-xfce_private_rnd4k_q1t1_write 3:write_bandwidth_kb: 1452.00 :green_circle: ( previous job: 1130.00, improvement: 128.50%)
  • fedora-41-xfce_volatile_seq1m_q8t1_read 3:read_bandwidth_kb: 356416.00 :small_red_triangle_up: ( previous job: 369868.00, degradation: 96.36%)
  • fedora-41-xfce_volatile_seq1m_q8t1_write 3:write_bandwidth_kb: 230456.00 :green_circle: ( previous job: 179949.00, improvement: 128.07%)
  • fedora-41-xfce_volatile_seq1m_q1t1_read 3:read_bandwidth_kb: 356173.00 :green_circle: ( previous job: 324737.00, improvement: 109.68%)
  • fedora-41-xfce_volatile_seq1m_q1t1_write 3:write_bandwidth_kb: 46638.00 :green_circle: ( previous job: 17567.00, improvement: 265.49%)
  • fedora-41-xfce_volatile_rnd4k_q32t1_read 3:read_bandwidth_kb: 75436.00 :small_red_triangle_up: ( previous job: 79021.00, degradation: 95.46%)
  • fedora-41-xfce_volatile_rnd4k_q1t1_read 3:read_bandwidth_kb: 8299.00 :green_circle: ( previous job: 7867.00, improvement: 105.49%)

@marmarek
Copy link
Member

I don't see any failure with this PR (but not QubesOS/qubes-core-qrexec#186): https://openqa.qubes-os.org/tests/129865

@marmarek
Copy link
Member

marmarek commented Apr 13, 2025

I finally got the correct stderr logged (#670, https://github.com/QubesOS/qubes-core-qrexec/pull/196/files) and got this:

2025-04-12 18:47:10.164 qrexec-fork-server[1073]: exec.c:165:exec_qubes_rpc2: execve: Exec format error

It's a progress I guess...

@marmarek
Copy link
Member

Ok, this is clearly bug in tests - they create rpc services without shebang, so it works only when called via shell - which not always is the case.

@marmarek
Copy link
Member

Fix: #675

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants