multi-rails hang io #10420

ivanallen · 2025-01-15T11:45:47Z

Describe the bug

When I use multi-rail, ucx_perf will hang. When the ucx_perf is specified 'UCX_NET_DEVICES=mlx5_bond_0:1' or 'UCX_NET_DEVICES=mlx5_bond_1:1', it works properly.

Steps to Reproduce

Command line

# server
UCX_PROTO_INFO=y ./install-release-mt/bin/ucx_perftest -e
# client
UCX_TLS=rc UCX_PROTO_INFO=y ./install-release-mt/bin/ucx_perftest 10.16.29.13 -t ucp_am_bw -s 1048576 -n 50000 -e

UCX version used: 1.18.0-rc3
Any UCX environment variables used

Setup and versions

OS version (e.g Linux distro) + CPU architecture (x86_64/aarch64/ppc64le/...)

Linux localhost.localdomain 5.14.0-162.nos.4.el8.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Nov 24 07:51:00 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

For RDMA/IB/RoCE related issues:
- Driver version:
  MLNX_OFED_LINUX-5.8-4.1.5.0
- HW information from ibstat or ibv_devinfo -vv command

[root@localhost ucx-1.18.0]# ibstat
CA 'mlx5_bond_0'
        CA type: MT4125
        Number of ports: 1
        Firmware version: 22.36.1010
        Hardware version: 0
        Node GUID: 0xa088c20300b487d6
        System image GUID: 0xa088c20300b487d6
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 100
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0xa288c2fffeb487d6
                Link layer: Ethernet
CA 'mlx5_bond_1'
        CA type: MT4125
        Number of ports: 1
        Firmware version: 22.36.1010
        Hardware version: 0
        Node GUID: 0xa088c20300b4b04a
        System image GUID: 0xa088c20300b4b04a
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 100
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0xa288c2fffeb4b04a
                Link layer: Ethernet


[root@localhost ucx-1.18.0]# ibv_devinfo -vv

                                        RAW_IP_CSUM
                                        MANAGED_FLOW_STEERING
                                        Unknown flags: 0xC8400000
        max_sge:                        30
        max_sge_rd:                     30
        max_cq:                         16777216
        max_cqe:                        4194303
        max_mr:                         16777216
        max_pd:                         8388608
        max_qp_rd_atom:                 16
        max_ee_rd_atom:                 0
        max_res_rd_atom:                2097152
        max_qp_init_rd_atom:            16
        max_ee_init_rd_atom:            0
        atomic_cap:                     ATOMIC_HCA (1)
        max_ee:                         0
        max_rdd:                        0
        max_mw:                         16777216
        max_raw_ipv6_qp:                0
        max_raw_ethy_qp:                0
        max_mcast_grp:                  2097152
        max_mcast_qp_attach:            240
        max_total_mcast_qp_attach:      503316480
        max_ah:                         2147483647
        max_fmr:                        0
        max_srq:                        8388608
        max_srq_wr:                     32767
        max_srq_sge:                    31
        max_pkeys:                      128
        local_ca_ack_delay:             16
        general_odp_caps:
                                        ODP_SUPPORT
                                        ODP_SUPPORT_IMPLICIT
        rc_odp_caps:
                                        SUPPORT_SEND
                                        SUPPORT_RECV
                                        SUPPORT_WRITE
                                        SUPPORT_READ
                                        SUPPORT_SRQ
        uc_odp_caps:
                                        NO SUPPORT
        ud_odp_caps:
                                        SUPPORT_SEND
        xrc_odp_caps:
                                        SUPPORT_SEND
                                        SUPPORT_WRITE
                                        SUPPORT_READ
                                        SUPPORT_SRQ
        completion timestamp_mask:                      0x7fffffffffffffff
        hca_core_clock:                 1000000kHZ
        raw packet caps:
                                        C-VLAN stripping offload
                                        Scatter FCS offload
                                        IP csum offload
                                        Delay drop
        device_cap_flags_ex:            0x30000054ED721C36
                                        RAW_SCATTER_FCS
                                        PCI_WRITE_END_PADDING
                                        Unknown flags: 0x3000004000000000
        tso_caps:
                max_tso:                        262144
                supported_qp:
                                        SUPPORT_RAW_PACKET
        rss_caps:
                max_rwq_indirection_tables:                     524288
                max_rwq_indirection_table_size:                 2048
                rx_hash_function:                               0x1
                rx_hash_fields_mask:                            0x800000FF
                supported_qp:
                                        SUPPORT_RAW_PACKET
        max_wq_type_rq:                 8388608
        packet_pacing_caps:
                qp_rate_limit_min:      1kbps
                qp_rate_limit_max:      100000000kbps
                supported_qp:
                                        SUPPORT_RAW_PACKET
        tag matching not supported

        cq moderation caps:
                max_cq_count:   65535
                max_cq_period:  4095 us

        maximum available device memory:        131072Bytes

        num_comp_vectors:               63
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet
                        max_msg_sz:             0x40000000
                        port_cap_flags:         0x04010000
                        port_cap_flags2:        0x0000
                        max_vl_num:             invalid value (0)
                        bad_pkey_cntr:          0x0
                        qkey_viol_cntr:         0x0
                        sm_sl:                  0
                        pkey_tbl_len:           1
                        gid_tbl_len:            255
                        subnet_timeout:         0
                        init_type_reply:        0
                        active_width:           4X (2)
                        active_speed:           25.0 Gbps (32)
                        phys_state:             LINK_UP (5)
                        GID[  0]:               fe80:0000:0000:0000:a288:c2ff:feb4:87d6, RoCE v1
                        GID[  1]:               fe80::a288:c2ff:feb4:87d6, RoCE v2
                        GID[  2]:               0000:0000:0000:0000:0000:ffff:0a10:1d0d, RoCE v1
                        GID[  3]:               ::ffff:10.16.29.13, RoCE v2

hca_id: mlx5_bond_1
        transport:                      InfiniBand (0)
        fw_ver:                         22.36.1010
        node_guid:                      a088:c203:00b4:b04a
        sys_image_guid:                 a088:c203:00b4:b04a
        vendor_id:                      0x02c9
        vendor_part_id:                 4125
        hw_ver:                         0x0

        max_srq_sge:                    31
        max_pkeys:                      128
        local_ca_ack_delay:             16
        general_odp_caps:
                                        ODP_SUPPORT
                                        ODP_SUPPORT_IMPLICIT
        rc_odp_caps:
                                        SUPPORT_SEND
                                        SUPPORT_RECV
                                        SUPPORT_WRITE
                                        SUPPORT_READ
                                        SUPPORT_SRQ
        uc_odp_caps:
                                        NO SUPPORT
        ud_odp_caps:
                                        SUPPORT_SEND
        xrc_odp_caps:
                                        SUPPORT_SEND
                                        SUPPORT_WRITE
                                        SUPPORT_READ
                                        SUPPORT_SRQ
        completion timestamp_mask:                      0x7fffffffffffffff
        hca_core_clock:                 1000000kHZ
        raw packet caps:
                                        C-VLAN stripping offload
                                        Scatter FCS offload
                                        IP csum offload
                                        Delay drop
        device_cap_flags_ex:            0x30000054ED721C36
                                        RAW_SCATTER_FCS
                                        PCI_WRITE_END_PADDING
                                        Unknown flags: 0x3000004000000000
        tso_caps:
                max_tso:                        262144
                supported_qp:
                                        SUPPORT_RAW_PACKET
        rss_caps:
                max_rwq_indirection_tables:                     524288
                max_rwq_indirection_table_size:                 2048
                rx_hash_function:                               0x1
                rx_hash_fields_mask:                            0x800000FF
                supported_qp:
                                        SUPPORT_RAW_PACKET
        max_wq_type_rq:                 8388608
        packet_pacing_caps:
                qp_rate_limit_min:      1kbps
                qp_rate_limit_max:      100000000kbps
                supported_qp:
                                        SUPPORT_RAW_PACKET
        tag matching not supported

        cq moderation caps:
                max_cq_count:   65535
                max_cq_period:  4095 us

        maximum available device memory:        131072Bytes

        num_comp_vectors:               63
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet
                        max_msg_sz:             0x40000000
                        port_cap_flags:         0x04010000
                        port_cap_flags2:        0x0000
                        max_vl_num:             invalid value (0)
                        bad_pkey_cntr:          0x0
                        qkey_viol_cntr:         0x0
                        sm_sl:                  0
                        pkey_tbl_len:           1
                        gid_tbl_len:            255
                        subnet_timeout:         0
                        init_type_reply:        0
                        active_width:           4X (2)
                        active_speed:           25.0 Gbps (32)
                        phys_state:             LINK_UP (5)
                        GID[  0]:               fe80:0000:0000:0000:a288:c2ff:feb4:b04a, RoCE v1
                        GID[  1]:               fe80::a288:c2ff:feb4:b04a, RoCE v2
                        GID[  2]:               0000:0000:0000:0000:0000:ffff:0a10:270d, RoCE v1
                        GID[  3]:               ::ffff:10.16.39.13, RoCE v2

Additional information (depending on the issue)

client

[root@localhost ucx-1.18.0]# show_gids
DEV     PORT    INDEX   GID                                     IPv4            VER     DEV
---     ----    -----   ---                                     ------------    ---     ---
mlx5_bond_0     1       0       fe80:0000:0000:0000:a288:c2ff:feb4:87d6                 v1      bond0
mlx5_bond_0     1       1       fe80:0000:0000:0000:a288:c2ff:feb4:87d6                 v2      bond0
mlx5_bond_0     1       2       0000:0000:0000:0000:0000:ffff:0a10:1d0d 10.16.29.13     v1      bond0
mlx5_bond_0     1       3       0000:0000:0000:0000:0000:ffff:0a10:1d0d 10.16.29.13     v2      bond0
mlx5_bond_1     1       0       fe80:0000:0000:0000:a288:c2ff:feb4:b04a                 v1      bond1
mlx5_bond_1     1       1       fe80:0000:0000:0000:a288:c2ff:feb4:b04a                 v2      bond1
mlx5_bond_1     1       2       0000:0000:0000:0000:0000:ffff:0a10:270d 10.16.39.13     v1      bond1
mlx5_bond_1     1       3       0000:0000:0000:0000:0000:ffff:0a10:270d 10.16.39.13     v2      bond1
n_gids_found=8

server

[root@localhost ucx-1.18.0]# show_gids
DEV     PORT    INDEX   GID                                     IPv4            VER     DEV
---     ----    -----   ---                                     ------------    ---     ---
mlx5_bond_0     1       0       fe80:0000:0000:0000:a288:c2ff:feb4:87e6                 v1      bond0
mlx5_bond_0     1       1       fe80:0000:0000:0000:a288:c2ff:feb4:87e6                 v2      bond0
mlx5_bond_0     1       2       0000:0000:0000:0000:0000:ffff:0a10:1d0c 10.16.29.12     v1      bond0
mlx5_bond_0     1       3       0000:0000:0000:0000:0000:ffff:0a10:1d0c 10.16.29.12     v2      bond0
mlx5_bond_1     1       0       fe80:0000:0000:0000:a288:c2ff:feb4:a563                 v1      bond1
mlx5_bond_1     1       1       fe80:0000:0000:0000:a288:c2ff:feb4:a563                 v2      bond1
mlx5_bond_1     1       2       0000:0000:0000:0000:0000:ffff:0a10:270c 10.16.39.12     v1      bond1
mlx5_bond_1     1       3       0000:0000:0000:0000:0000:ffff:0a10:270c 10.16.39.12     v2      bond1
n_gids_found=8

client log:

ucx.log.txt

The text was updated successfully, but these errors were encountered:

ivanallen · 2025-01-15T11:48:51Z

[root@localhost ucx-1.18.0]# ./install-release-mt/bin/ucx_info -d
#
# Memory domain: self
#     Component: self
#             register: unlimited, cost: 0 nsec
#           remote key: 0 bytes
#           rkey_ptr is supported
#         memory types: host (access,reg_nonblock,reg,cache)
#
#      Transport: self
#         Device: memory
#           Type: loopback
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 19360.00 MB/sec
#              latency: 0 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 8K
#             am_bcopy: <= 8K
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 0 bytes
#        iface address: 8 bytes
#       error handling: ep_check
#
#
# Memory domain: tcp
#     Component: tcp
#             register: unlimited, cost: 0 nsec
#           remote key: 0 bytes
#         memory types: host (access,reg_nonblock,reg,cache)
#
#      Transport: tcp
#         Device: bond0
#           Type: network
#  System device: bond0 (0)
#
#      capabilities:
#            bandwidth: 2200.00/ppn + 0.00 MB/sec
#              latency: 5203 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 1
#     device num paths: 1
#              max eps: 256
#       device address: 6 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#
#      Transport: tcp
#         Device: bond1
#           Type: network
#  System device: bond1 (1)
#
#      capabilities:
#            bandwidth: 2200.00/ppn + 0.00 MB/sec
#              latency: 5203 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 1
#     device num paths: 1
#              max eps: 256
#       device address: 6 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#
#      Transport: tcp
#         Device: ens16f0
#           Type: network
#  System device: ens16f0 (2)
#
#      capabilities:
#            bandwidth: 113.16/ppn + 0.00 MB/sec
#              latency: 5776 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 0
#     device num paths: 1
#              max eps: 256
#       device address: 6 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#
#      Transport: tcp
#         Device: lo
#           Type: network
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 11.91/ppn + 0.00 MB/sec
#              latency: 10960 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 1
#     device num paths: 1
#              max eps: 256
#       device address: 18 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#
#
# Connection manager: tcp
#      max_conn_priv: 2064 bytes
#
# Memory domain: sysv
#     Component: sysv
#             allocate: unlimited
#           remote key: 12 bytes
#           rkey_ptr is supported
#         memory types: host (access,alloc,cache)
#
#      Transport: sysv
#         Device: memory
#           Type: intra-node
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 15360.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 8 bytes
#        iface address: 8 bytes
#       error handling: ep_check
#
#
# Memory domain: posix
#     Component: posix
#             allocate: <= 263730384K
#           remote key: 24 bytes
#           rkey_ptr is supported
#         memory types: host (access,alloc,cache)
#
#      Transport: posix
#         Device: memory
#           Type: intra-node
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 15360.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 8 bytes
#        iface address: 8 bytes
#       error handling: ep_check
#
#
# Memory domain: mlx5_bond_1
#     Component: ib
#             register: unlimited, dmabuf, cost: 16000 + 0.060 * N nsec
#           remote key: 8 bytes
#           local memory handle is required for zcopy
#           memory invalidation is supported
#         memory types: host (access,reg,cache)
#
#      Transport: rc_verbs
#         Device: mlx5_bond_1:1
#           Type: network
#  System device: mlx5_bond_1 (1)
#
#      capabilities:
#            bandwidth: 21915.68/ppn + 0.00 MB/sec
#              latency: 800 + 1.000 * N nsec
#             overhead: 75 nsec
#            put_short: <= 124
#            put_bcopy: <= 8256
#            put_zcopy: <= 1G, up to 5 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 1K
#            get_bcopy: <= 8256
#            get_zcopy: 65..1G, up to 5 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 1K
#             am_short: <= 123
#             am_bcopy: <= 8255
#             am_zcopy: <= 8255, up to 4 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 127
#               domain: device
#           atomic_add: 64 bit
#          atomic_fadd: 64 bit
#         atomic_cswap: 64 bit
#           connection: to ep
#      device priority: 60
#     device num paths: 2
#              max eps: 256
#       device address: 18 bytes
#           ep address: 7 bytes
#       error handling: peer failure, ep_check
#
#
#      Transport: ud_verbs
#         Device: mlx5_bond_1:1
#           Type: network
#  System device: mlx5_bond_1 (1)
#
#      capabilities:
#            bandwidth: 21915.68/ppn + 0.00 MB/sec
#              latency: 830 nsec
#             overhead: 105 nsec
#             am_short: <= 116
#             am_bcopy: <= 1016
#             am_zcopy: <= 1016, up to 5 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 920
#           connection: to ep, to iface
#      device priority: 60
#     device num paths: 2
#              max eps: inf
#       device address: 18 bytes
#        iface address: 3 bytes
#           ep address: 6 bytes
#       error handling: peer failure, ep_check
#
#
#      Transport: dc_mlx5
#         Device: mlx5_bond_1:1
#           Type: network
#  System device: mlx5_bond_1 (1)
#
#      capabilities:
#            bandwidth: 21915.68/ppn + 0.00 MB/sec
#              latency: 860 nsec
#             overhead: 40 nsec
#            put_short: <= 2K
#            put_bcopy: <= 8256
#            put_zcopy: <= 1G, up to 11 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 1K
#            get_bcopy: <= 8256
#            get_zcopy: 65..1G, up to 11 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 1K
#             am_short: <= 2046
#             am_bcopy: <= 8254
#             am_zcopy: <= 8254, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 138
#               domain: device
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 60
#     device num paths: 2
#              max eps: inf
#       device address: 18 bytes
#        iface address: 7 bytes
#       error handling: buffer (zcopy), remote access, peer failure
#
#
#      Transport: rc_mlx5
#         Device: mlx5_bond_1:1
#           Type: network
#  System device: mlx5_bond_1 (1)
#
#      capabilities:
#            bandwidth: 21915.68/ppn + 0.00 MB/sec
#              latency: 800 + 1.000 * N nsec
#             overhead: 40 nsec
#            put_short: <= 2K
#            put_bcopy: <= 8256
#            put_zcopy: <= 1G, up to 14 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 1K
#            get_bcopy: <= 8256
#            get_zcopy: 65..1G, up to 14 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 1K
#             am_short: <= 2046
#             am_bcopy: <= 8254
#             am_zcopy: <= 8254, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 186
#               domain: device
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to ep
#      device priority: 60
#     device num paths: 2
#              max eps: 256
#       device address: 18 bytes
#           ep address: 10 bytes
#       error handling: buffer (zcopy), remote access, peer failure, ep_check
#
#
#      Transport: ud_mlx5
#         Device: mlx5_bond_1:1
#           Type: network
#  System device: mlx5_bond_1 (1)
#
#      capabilities:
#            bandwidth: 21915.68/ppn + 0.00 MB/sec
#              latency: 830 nsec
#             overhead: 80 nsec
#             am_short: <= 180
#             am_bcopy: <= 1016
#             am_zcopy: <= 1016, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 132
#           connection: to ep, to iface
#      device priority: 60
#     device num paths: 2
#              max eps: inf
#       device address: 18 bytes
#        iface address: 3 bytes
#           ep address: 6 bytes
#       error handling: peer failure, ep_check
#
#
# Memory domain: mlx5_bond_0
#     Component: ib
#             register: unlimited, dmabuf, cost: 16000 + 0.060 * N nsec
#           remote key: 8 bytes
#           local memory handle is required for zcopy
#           memory invalidation is supported
#         memory types: host (access,reg,cache)
#
#      Transport: rc_verbs
#         Device: mlx5_bond_0:1
#           Type: network
#  System device: mlx5_bond_0 (0)
#
#      capabilities:
#            bandwidth: 21915.68/ppn + 0.00 MB/sec
#              latency: 800 + 1.000 * N nsec
#             overhead: 75 nsec
#            put_short: <= 124
#            put_bcopy: <= 8256
#            put_zcopy: <= 1G, up to 5 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 1K
#            get_bcopy: <= 8256
#            get_zcopy: 65..1G, up to 5 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 1K
#             am_short: <= 123
#             am_bcopy: <= 8255
#             am_zcopy: <= 8255, up to 4 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 127
#               domain: device
#           atomic_add: 64 bit
#          atomic_fadd: 64 bit
#         atomic_cswap: 64 bit
#           connection: to ep
#      device priority: 60
#     device num paths: 2
#              max eps: 256
#       device address: 18 bytes
#           ep address: 7 bytes
#       error handling: peer failure, ep_check
#
#
#      Transport: ud_verbs
#         Device: mlx5_bond_0:1
#           Type: network
#  System device: mlx5_bond_0 (0)
#
#      capabilities:
#            bandwidth: 21915.68/ppn + 0.00 MB/sec
#              latency: 830 nsec
#             overhead: 105 nsec
#             am_short: <= 116
#             am_bcopy: <= 1016
#             am_zcopy: <= 1016, up to 5 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 920
#           connection: to ep, to iface
#      device priority: 60
#     device num paths: 2
#              max eps: inf
#       device address: 18 bytes
#        iface address: 3 bytes
#           ep address: 6 bytes
#       error handling: peer failure, ep_check
#
#
#      Transport: dc_mlx5
#         Device: mlx5_bond_0:1
#           Type: network
#  System device: mlx5_bond_0 (0)
#
#      capabilities:
#            bandwidth: 21915.68/ppn + 0.00 MB/sec
#              latency: 860 nsec
#             overhead: 40 nsec
#            put_short: <= 2K
#            put_bcopy: <= 8256
#            put_zcopy: <= 1G, up to 11 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 1K
#            get_bcopy: <= 8256
#            get_zcopy: 65..1G, up to 11 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 1K
#             am_short: <= 2046
#             am_bcopy: <= 8254
#             am_zcopy: <= 8254, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 138
#               domain: device
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 60
#     device num paths: 2
#              max eps: inf
#       device address: 18 bytes
#        iface address: 7 bytes
#       error handling: buffer (zcopy), remote access, peer failure
#
#
#      Transport: rc_mlx5
#         Device: mlx5_bond_0:1
#           Type: network
#  System device: mlx5_bond_0 (0)
#
#      capabilities:
#            bandwidth: 21915.68/ppn + 0.00 MB/sec
#              latency: 800 + 1.000 * N nsec
#             overhead: 40 nsec
#            put_short: <= 2K
#            put_bcopy: <= 8256
#            put_zcopy: <= 1G, up to 14 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 1K
#            get_bcopy: <= 8256
#            get_zcopy: 65..1G, up to 14 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 1K
#             am_short: <= 2046
#             am_bcopy: <= 8254
#             am_zcopy: <= 8254, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 186
#               domain: device
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to ep
#      device priority: 60
#     device num paths: 2
#              max eps: 256
#       device address: 18 bytes
#           ep address: 10 bytes
#       error handling: buffer (zcopy), remote access, peer failure, ep_check
#
#
#      Transport: ud_mlx5
#         Device: mlx5_bond_0:1
#           Type: network
#  System device: mlx5_bond_0 (0)
#
#      capabilities:
#            bandwidth: 21915.68/ppn + 0.00 MB/sec
#              latency: 830 nsec
#             overhead: 80 nsec
#             am_short: <= 180
#             am_bcopy: <= 1016
#             am_zcopy: <= 1016, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 1K
#            am header: <= 132
#           connection: to ep, to iface
#      device priority: 60
#     device num paths: 2
#              max eps: inf
#       device address: 18 bytes
#        iface address: 3 bytes
#           ep address: 6 bytes
#       error handling: peer failure, ep_check
#
#
# Memory domain: mlx5_bond_1
#     Component: gga
#             register: unlimited, dmabuf, cost: 16000 + 0.060 * N nsec
#           remote key: 8 bytes
#           local memory handle is required for zcopy
#           memory invalidation is supported
#         memory types: host (access,reg,cache)
#   < no supported devices found >
#
# Memory domain: mlx5_bond_0
#     Component: gga
#             register: unlimited, dmabuf, cost: 16000 + 0.060 * N nsec
#           remote key: 8 bytes
#           local memory handle is required for zcopy
#           memory invalidation is supported
#         memory types: host (access,reg,cache)
#   < no supported devices found >
#
# Connection manager: rdmacm
#      max_conn_priv: 54 bytes
#
# Memory domain: cma
#     Component: cma
#             register: unlimited, cost: 9 nsec
#         memory types: host (access,reg_nonblock,reg,cache)
#
#      Transport: cma
#         Device: memory
#           Type: intra-node
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 11145.00 MB/sec
#              latency: 80 nsec
#             overhead: 2000 nsec
#            put_zcopy: unlimited, up to 16 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 1
#            get_zcopy: unlimited, up to 16 iov
#  get_opt_zcopy_align: <= 1
#        get_align_mtu: <= 1
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 8 bytes
#        iface address: 4 bytes
#       error handling: peer failure, ep_check
#

yosefe · 2025-01-15T14:09:51Z

@ivanallen can you pls post the output with UCX_PROTO_INFO=y and multi-rail?

ivanallen · 2025-01-16T01:39:13Z

@ivanallen can you pls post the output with UCX_PROTO_INFO=y and multi-rail?

@yosefe Hi, for detail log with UCX_LOG_LEVEL=data, see ucx.log.txt.

[root@localhost ucx-1.18.0]# UCX_TLS=rc UCX_PROTO_INFO=y ./install-release-mt/bin/ucx_perftest 10.16.29.13 -t ucp_am_bw -s 1048576 -n 50000 -e
[1736991281.802773] [localhost:42638:0]        perftest.c:800  UCX  WARN  CPU affinity is not set (bound to 96 cpus). Performance may be impacted.
+--------------+--------------+------------------------------+---------------------+-----------------------+
|              |              |       overhead (usec)        |   bandwidth (MB/s)  |  message rate (msg/s) |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
|    Stage     | # iterations | 50.0%ile | average | overall |  average |  overall |  average  |  overall  |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
[1736991281.943478] [localhost:42638:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1736991281.943488] [localhost:42638:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* from host memory                                                                       |
[1736991281.943493] [localhost:42638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.943496] [localhost:42638:0]   |                   0..2038 | short                                     | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1736991281.943499] [localhost:42638:0]   |                2039..8246 | copy-in                                   | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1736991281.943502] [localhost:42638:0]   |               8247..27188 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1736991281.943505] [localhost:42638:0]   |                27189..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_1:1/path0 and 50% on rc_mlx5/mlx5_bond_0:1/path0 |
[1736991281.943508] [localhost:42638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.943691] [localhost:42638:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1736991281.943695] [localhost:42638:0]   | perftest inter-node cfg#0 | active message by ucp_am_send*(fast-completion) from host memory                                                      |
[1736991281.943698] [localhost:42638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.943701] [localhost:42638:0]   |                   0..2038 | short                                     | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1736991281.943704] [localhost:42638:0]   |                2039..8246 | copy-in                                   | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1736991281.943707] [localhost:42638:0]   |               8247..13104 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1736991281.943708] [localhost:42638:0]   |             13105..262143 | multi-frag zero-copy                      | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1736991281.943710] [localhost:42638:0]   |                 256K..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_1:1/path0 and 50% on rc_mlx5/mlx5_bond_0:1/path0 |
[1736991281.943714] [localhost:42638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.944113] [localhost:42638:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1736991281.944117] [localhost:42638:0]   | perftest inter-node cfg#0 | active message by ucp_am_send*(multi) from host memory                                                                |
[1736991281.944121] [localhost:42638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.944123] [localhost:42638:0]   |                    0..514 | short                                     | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1736991281.944126] [localhost:42638:0]   |                 515..4844 | zero-copy                                 | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1736991281.944129] [localhost:42638:0]   |                 4845..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_1:1/path0 and 50% on rc_mlx5/mlx5_bond_0:1/path0 |
[1736991281.944132] [localhost:42638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.944301] [localhost:42638:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1736991281.944304] [localhost:42638:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* with reply flag from host memory                                                       |
[1736991281.944307] [localhost:42638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.944309] [localhost:42638:0]   |                   0..2030 | short                                     | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1736991281.944312] [localhost:42638:0]   |                2031..8238 | copy-in                                   | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1736991281.944316] [localhost:42638:0]   |               8239..27188 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1736991281.944319] [localhost:42638:0]   |                27189..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_1:1/path0 and 50% on rc_mlx5/mlx5_bond_0:1/path0 |
[1736991281.944322] [localhost:42638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.944472] [localhost:42638:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1736991281.944475] [localhost:42638:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* with reply flag(fast-completion) from host memory                                      |
[1736991281.944478] [localhost:42638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.944481] [localhost:42638:0]   |                   0..2030 | short                                     | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1736991281.944485] [localhost:42638:0]   |                2031..8238 | copy-in                                   | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1736991281.944488] [localhost:42638:0]   |               8239..13104 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1736991281.944490] [localhost:42638:0]   |             13105..262143 | multi-frag zero-copy                      | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1736991281.944493] [localhost:42638:0]   |                 256K..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_1:1/path0 and 50% on rc_mlx5/mlx5_bond_0:1/path0 |
[1736991281.944494] [localhost:42638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.944682] [localhost:42638:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1736991281.944686] [localhost:42638:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* with reply flag(multi) from host memory                                                |
[1736991281.944688] [localhost:42638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.944690] [localhost:42638:0]   |                    0..514 | short                                     | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1736991281.944693] [localhost:42638:0]   |                 515..4844 | zero-copy                                 | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1736991281.944696] [localhost:42638:0]   |                 4845..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_1:1/path0 and 50% on rc_mlx5/mlx5_bond_0:1/path0 |
[1736991281.944699] [localhost:42638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991315.743147] [localhost:42638:0]         libperf.c:1069 UCX  ERROR error handler called with status -80 (Endpoint timeout)

[1736991279.874886] [localhost:147696:0]        perftest.c:800  UCX  WARN  CPU affinity is not set (bound to 96 cpus). Performance may be impacted.
Waiting for connection...
Accepted connection from 10.16.29.12:53500
+----------------------------------------------------------------------------------------------------------+
| API:          protocol layer                                                                             |
| Test:         am bandwidth / message rate                                                                |
| Data layout:  (automatic)                                                                                |
| Send memory:  host                                                                                       |
| Recv memory:  host                                                                                       |
| Message size: 1048576                                                                                    |
| Window size:  32                                                                                         |
| AM header size: 0                                                                                        |
+----------------------------------------------------------------------------------------------------------+
[1736991281.946514] [localhost:147696:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1736991281.946526] [localhost:147696:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* from host memory                                                                       |
[1736991281.946532] [localhost:147696:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.946535] [localhost:147696:0]   |                   0..2038 | short                                     | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1736991281.946538] [localhost:147696:0]   |                2039..8246 | copy-in                                   | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1736991281.946542] [localhost:147696:0]   |               8247..27188 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1736991281.946546] [localhost:147696:0]   |                27189..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_0:1/path0 and 50% on rc_mlx5/mlx5_bond_1:1/path0 |
[1736991281.946549] [localhost:147696:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.946714] [localhost:147696:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1736991281.946717] [localhost:147696:0]   | perftest inter-node cfg#0 | active message by ucp_am_send*(fast-completion) from host memory                                                      |
[1736991281.946719] [localhost:147696:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.946722] [localhost:147696:0]   |                   0..2038 | short                                     | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1736991281.946724] [localhost:147696:0]   |                2039..8246 | copy-in                                   | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1736991281.946726] [localhost:147696:0]   |               8247..13104 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1736991281.946727] [localhost:147696:0]   |             13105..262143 | multi-frag zero-copy                      | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1736991281.946731] [localhost:147696:0]   |                 256K..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_0:1/path0 and 50% on rc_mlx5/mlx5_bond_1:1/path0 |
[1736991281.946734] [localhost:147696:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.947121] [localhost:147696:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1736991281.947125] [localhost:147696:0]   | perftest inter-node cfg#0 | active message by ucp_am_send*(multi) from host memory                                                                |
[1736991281.947127] [localhost:147696:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.947130] [localhost:147696:0]   |                    0..514 | short                                     | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1736991281.947133] [localhost:147696:0]   |                 515..4844 | zero-copy                                 | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1736991281.947136] [localhost:147696:0]   |                 4845..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_0:1/path0 and 50% on rc_mlx5/mlx5_bond_1:1/path0 |
[1736991281.947139] [localhost:147696:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.947302] [localhost:147696:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1736991281.947305] [localhost:147696:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* with reply flag from host memory                                                       |
[1736991281.947307] [localhost:147696:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.947310] [localhost:147696:0]   |                   0..2030 | short                                     | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1736991281.947312] [localhost:147696:0]   |                2031..8238 | copy-in                                   | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1736991281.947315] [localhost:147696:0]   |               8239..27188 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1736991281.947318] [localhost:147696:0]   |                27189..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_0:1/path0 and 50% on rc_mlx5/mlx5_bond_1:1/path0 |
[1736991281.947321] [localhost:147696:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.947465] [localhost:147696:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1736991281.947468] [localhost:147696:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* with reply flag(fast-completion) from host memory                                      |
[1736991281.947470] [localhost:147696:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.947474] [localhost:147696:0]   |                   0..2030 | short                                     | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1736991281.947477] [localhost:147696:0]   |                2031..8238 | copy-in                                   | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1736991281.947479] [localhost:147696:0]   |               8239..13104 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1736991281.947481] [localhost:147696:0]   |             13105..262143 | multi-frag zero-copy                      | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1736991281.947483] [localhost:147696:0]   |                 256K..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_0:1/path0 and 50% on rc_mlx5/mlx5_bond_1:1/path0 |
[1736991281.947486] [localhost:147696:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.947667] [localhost:147696:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1736991281.947670] [localhost:147696:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* with reply flag(multi) from host memory                                                |
[1736991281.947672] [localhost:147696:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991281.947676] [localhost:147696:0]   |                    0..514 | short                                     | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1736991281.947678] [localhost:147696:0]   |                 515..4844 | zero-copy                                 | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1736991281.947681] [localhost:147696:0]   |                 4845..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_0:1/path0 and 50% on rc_mlx5/mlx5_bond_1:1/path0 |
[1736991281.947685] [localhost:147696:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1736991315.743435] [localhost:147696:0]         libperf.c:1069 UCX  ERROR error handler called with status -80 (Endpoint timeout)
[root@localhost ucx-1.18.0]#

ucx.log.txt

yosefe · 2025-01-16T09:13:46Z

Can you pls try adding UCX_IB_ROCE_LOCAL_SUBNET=y?

ivanallen · 2025-01-16T10:32:12Z

Can you pls try adding UCX_IB_ROCE_LOCAL_SUBNET=y?

@yosefe When I added this configuration, the client crashed.

[root@node12 ucx-1.18.0]# UCX_IB_ROCE_LOCAL_SUBNET=y UCX_PROTO_ENABLE=y UCX_TLS=rc  UCX_PROTO_INFO=y ./install-release-mt/bin/ucx_perftest 10.16.29.13 -t ucp_am_bw -s 1048576  -n 5000000 -e
[1737023435.554769] [node13:2664638:0]        perftest.c:800  UCX  WARN  CPU affinity is not set (bound to 96 cpus). Performance may be impacted.
+--------------+--------------+------------------------------+---------------------+-----------------------+
|              |              |       overhead (usec)        |   bandwidth (MB/s)  |  message rate (msg/s) |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
|    Stage     | # iterations | 50.0%ile | average | overall |  average |  overall |  average  |  overall  |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
[1737023435.700071] [node13:2664638:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1737023435.700082] [node13:2664638:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* from host memory                                                                       |
[1737023435.700086] [node13:2664638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737023435.700089] [node13:2664638:0]   |                   0..2038 | short                                     | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737023435.700092] [node13:2664638:0]   |                2039..8246 | copy-in                                   | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737023435.700094] [node13:2664638:0]   |               8247..27188 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737023435.700098] [node13:2664638:0]   |                27189..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_1:1/path0 and 50% on rc_mlx5/mlx5_bond_0:1/path0 |
[1737023435.700101] [node13:2664638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737023435.700268] [node13:2664638:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1737023435.700272] [node13:2664638:0]   | perftest inter-node cfg#0 | active message by ucp_am_send*(fast-completion) from host memory                                                      |
[1737023435.700274] [node13:2664638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737023435.700278] [node13:2664638:0]   |                   0..2038 | short                                     | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737023435.700281] [node13:2664638:0]   |                2039..8246 | copy-in                                   | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737023435.700283] [node13:2664638:0]   |               8247..13104 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737023435.700286] [node13:2664638:0]   |             13105..262143 | multi-frag zero-copy                      | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737023435.700289] [node13:2664638:0]   |                 256K..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_1:1/path0 and 50% on rc_mlx5/mlx5_bond_0:1/path0 |
[1737023435.700292] [node13:2664638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737023435.700681] [node13:2664638:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1737023435.700685] [node13:2664638:0]   | perftest inter-node cfg#0 | active message by ucp_am_send*(multi) from host memory                                                                |
[1737023435.700688] [node13:2664638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737023435.700691] [node13:2664638:0]   |                    0..514 | short                                     | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737023435.700695] [node13:2664638:0]   |                 515..4844 | zero-copy                                 | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737023435.700698] [node13:2664638:0]   |                 4845..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_1:1/path0 and 50% on rc_mlx5/mlx5_bond_0:1/path0 |
[1737023435.700699] [node13:2664638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737023435.700858] [node13:2664638:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1737023435.700861] [node13:2664638:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* with reply flag from host memory                                                       |
[1737023435.700863] [node13:2664638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737023435.700867] [node13:2664638:0]   |                   0..2030 | short                                     | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737023435.700870] [node13:2664638:0]   |                2031..8238 | copy-in                                   | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737023435.700873] [node13:2664638:0]   |               8239..27188 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737023435.700875] [node13:2664638:0]   |                27189..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_1:1/path0 and 50% on rc_mlx5/mlx5_bond_0:1/path0 |
[1737023435.700877] [node13:2664638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737023435.701020] [node13:2664638:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1737023435.701024] [node13:2664638:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* with reply flag(fast-completion) from host memory                                      |
[1737023435.701025] [node13:2664638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737023435.701028] [node13:2664638:0]   |                   0..2030 | short                                     | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737023435.701031] [node13:2664638:0]   |                2031..8238 | copy-in                                   | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737023435.701035] [node13:2664638:0]   |               8239..13104 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737023435.701039] [node13:2664638:0]   |             13105..262143 | multi-frag zero-copy                      | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737023435.701040] [node13:2664638:0]   |                 256K..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_1:1/path0 and 50% on rc_mlx5/mlx5_bond_0:1/path0 |
[1737023435.701042] [node13:2664638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737023435.701217] [node13:2664638:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1737023435.701221] [node13:2664638:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* with reply flag(multi) from host memory                                                |
[1737023435.701223] [node13:2664638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737023435.701226] [node13:2664638:0]   |                    0..514 | short                                     | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737023435.701229] [node13:2664638:0]   |                 515..4844 | zero-copy                                 | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737023435.701231] [node13:2664638:0]   |                 4845..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_1:1/path0 and 50% on rc_mlx5/mlx5_bond_0:1/path0 |
[1737023435.701232] [node13:2664638:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737023435.708458] [node13:2664638:0]          wireup.c:1214 UCX  ERROR   old: am_lane 0 wireup_msg_lane 4 cm_lane <none> keepalive_lane 4 reachable_mds 0x30
[1737023435.708472] [node13:2664638:0]          wireup.c:1237 UCX  ERROR   old: lane[0]:  2:rc_mlx5/mlx5_bond_1:1.0 md[0]  -> md[5]/ib/sysdev[1] seg 4294967295 rma_bw#1 am am_bw#0
[1737023435.708478] [node13:2664638:0]          wireup.c:1237 UCX  ERROR   old: lane[1]:  6:rc_mlx5/mlx5_bond_0:1.0 md[1]  -> md[4]/ib/sysdev[0] seg 4294967295 rma_bw#0
[1737023435.708483] [node13:2664638:0]          wireup.c:1237 UCX  ERROR   old: lane[2]:  2:rc_mlx5/mlx5_bond_1:1.1 md[0]  -> md[5]/ib/sysdev[1] seg 4294967295 rma_bw#3
[1737023435.708487] [node13:2664638:0]          wireup.c:1237 UCX  ERROR   old: lane[3]:  6:rc_mlx5/mlx5_bond_0:1.1 md[1]  -> md[4]/ib/sysdev[0] seg 4294967295 rma_bw#2
[1737023435.708492] [node13:2664638:0]          wireup.c:1237 UCX  ERROR   old: lane[4]:  3:ud_mlx5/mlx5_bond_1:1.0 md[0]  -> md[5]/ib/sysdev[1] seg 4294967295 keepalive wireup
[1737023435.708495] [node13:2664638:0]          wireup.c:1240 UCX  ERROR   old: err mode 1, flags 0x0
[1737023435.708499] [node13:2664638:0]          wireup.c:1214 UCX  ERROR   new: am_lane 0 wireup_msg_lane 3 cm_lane <none> keepalive_lane 0 reachable_mds 0x30
[1737023435.708503] [node13:2664638:0]          wireup.c:1237 UCX  ERROR   new: lane[0]:  2:rc_mlx5/mlx5_bond_1:1.0 md[0]  -> md[5]/ib/sysdev[1] seg 4294967295 rma_bw#1 am am_bw#0 keepalive
[1737023435.708507] [node13:2664638:0]          wireup.c:1237 UCX  ERROR   new: lane[1]:  6:rc_mlx5/mlx5_bond_0:1.0 md[1]  -> md[4]/ib/sysdev[0] seg 4294967295 rma_bw#0
[1737023435.708511] [node13:2664638:0]          wireup.c:1237 UCX  ERROR   new: lane[2]:  2:rc_mlx5/mlx5_bond_1:1.1 md[0]  -> md[5]/ib/sysdev[1] seg 4294967295 rma_bw#3
[1737023435.708515] [node13:2664638:0]          wireup.c:1237 UCX  ERROR   new: lane[3]:  6:rc_mlx5/mlx5_bond_0:1.1 md[1]  -> md[4]/ib/sysdev[0] seg 4294967295 rma_bw#2 wireup
[1737023435.708519] [node13:2664638:0]          wireup.c:1240 UCX  ERROR   new: err mode 1, flags 0x0
[node13:2664638:0:2664638]      wireup.c:1734 Fatal: endpoint reconfiguration not supported yet
==== backtrace (tid:2664638) ====
 0  /root/liufeng/ucx-1.18.0/install-release-mt/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f8595f99e24]
 1  /root/liufeng/ucx-1.18.0/install-release-mt/lib/libucs.so.0(ucs_fatal_error_message+0xb2) [0x7f8595f96fd2]
 2  /root/liufeng/ucx-1.18.0/install-release-mt/lib/libucs.so.0(+0x300a9) [0x7f8595f970a9]
 3  /root/liufeng/ucx-1.18.0/install-release-mt/lib/libucp.so.0(ucp_wireup_init_lanes+0x1190) [0x7f85960c7cc0]
 4  /root/liufeng/ucx-1.18.0/install-release-mt/lib/libucp.so.0(+0xb8e69) [0x7f85960c7e69]
 5  /root/liufeng/ucx-1.18.0/install-release-mt/lib/libucp.so.0(+0xb9711) [0x7f85960c8711]
 6  /root/liufeng/ucx-1.18.0/install-release-mt/lib/ucx/libuct_ib.so.0(uct_ud_ep_process_rx+0x22f) [0x7f8594a9e07f]
 7  /root/liufeng/ucx-1.18.0/install-release-mt/lib/ucx/libuct_ib_mlx5.so.0(+0x50284) [0x7f8594398284]
 8  /root/liufeng/ucx-1.18.0/install-release-mt/lib/ucx/libuct_ib_mlx5.so.0(+0x4bd0b) [0x7f8594393d0b]
 9  /root/liufeng/ucx-1.18.0/install-release-mt/lib/libucs.so.0(__ucs_async_poll_missed+0x116) [0x7f8595f81606]
10  /root/liufeng/ucx-1.18.0/install-release-mt/lib/libucp.so.0(ucp_worker_progress+0x138) [0x7f859605f8d8]
11  ./install-release-mt/bin/ucx_perftest() [0x40b4cf]
12  ./install-release-mt/bin/ucx_perftest() [0x40a399]
13  ./install-release-mt/bin/ucx_perftest() [0x406b20]
14  ./install-release-mt/bin/ucx_perftest() [0x406ecf]
15  ./install-release-mt/bin/ucx_perftest() [0x404b47]
16  /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f8594f5eca3]
17  ./install-release-mt/bin/ucx_perftest() [0x404d3e]
=================================
[node13:2664638:0:2664638] Process frozen, press Enter to attach a debugger...

yosefe · 2025-01-16T10:44:09Z

@ivanallen can you pls set the same UCX env vars for the sever and the client?
Seems UCX_TLS=rc is set for the client but not for the server which leads to difference in ud lane selection.

ivanallen · 2025-01-16T11:08:18Z

@yosefe

In this environment, I have executed ifdown bond0; sleep 1; ifup bond0. After that, ucx_perf cannot be executed successfully.

[root@node12 ucx-1.18.0]# show_gids
DEV     PORT    INDEX   GID                                     IPv4            VER     DEV
---     ----    -----   ---                                     ------------    ---     ---
mlx5_bond_0     1       0       fe80:0000:0000:0000:a288:c2ff:feb4:87e6                 v1      bond0
mlx5_bond_0     1       1       fe80:0000:0000:0000:a288:c2ff:feb4:87e6                 v2      bond0
mlx5_bond_0     1       2       0000:0000:0000:0000:0000:ffff:0a10:1d0c 10.16.29.12     v1      bond0
mlx5_bond_0     1       3       0000:0000:0000:0000:0000:ffff:0a10:1d0c 10.16.29.12     v2      bond0
mlx5_bond_1     1       0       fe80:0000:0000:0000:a288:c2ff:feb4:a562                 v1      bond1
mlx5_bond_1     1       1       fe80:0000:0000:0000:a288:c2ff:feb4:a562                 v2      bond1
mlx5_bond_1     1       2       0000:0000:0000:0000:0000:ffff:0a10:270c 10.16.39.12     v1      bond1
mlx5_bond_1     1       3       0000:0000:0000:0000:0000:ffff:0a10:270c 10.16.39.12     v2      bond1
n_gids_found=8

server:

[root@node13 ucx-1.18.0]# UCX_IB_ROCE_LOCAL_SUBNET=y UCX_PROTO_ENABLE=y UCX_TLS=rc UCX_PROTO_INFO=y ./install-release-mt/bin/ucx_perftest
[1737025505.837844] [node13:2857558:0]        perftest.c:800  UCX  WARN  CPU affinity is not set (bound to 96 cpus). Performance may be impacted.
Waiting for connection...
Accepted connection from 10.16.29.12:43062
+----------------------------------------------------------------------------------------------------------+
| API:          protocol layer                                                                             |
| Test:         am bandwidth / message rate                                                                |
| Data layout:  (automatic)                                                                                |
| Send memory:  host                                                                                       |
| Recv memory:  host                                                                                       |
| Message size: 1048576                                                                                    |
| Window size:  32                                                                                         |
| AM header size: 0                                                                                        |
+----------------------------------------------------------------------------------------------------------+
[1737025510.219679] [node13:2857558:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1737025510.219690] [node13:2857558:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* from host memory                                                                       |
[1737025510.219694] [node13:2857558:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.219697] [node13:2857558:0]   |                   0..2038 | short                                     | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1737025510.219701] [node13:2857558:0]   |                2039..8246 | copy-in                                   | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1737025510.219703] [node13:2857558:0]   |               8247..27188 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1737025510.219706] [node13:2857558:0]   |                27189..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_0:1/path0 and 50% on rc_mlx5/mlx5_bond_1:1/path0 |
[1737025510.219709] [node13:2857558:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.219887] [node13:2857558:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1737025510.219890] [node13:2857558:0]   | perftest inter-node cfg#0 | active message by ucp_am_send*(fast-completion) from host memory                                                      |
[1737025510.219892] [node13:2857558:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.219895] [node13:2857558:0]   |                   0..2038 | short                                     | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1737025510.219898] [node13:2857558:0]   |                2039..8246 | copy-in                                   | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1737025510.219901] [node13:2857558:0]   |               8247..13104 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1737025510.219904] [node13:2857558:0]   |             13105..262143 | multi-frag zero-copy                      | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1737025510.219906] [node13:2857558:0]   |                 256K..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_0:1/path0 and 50% on rc_mlx5/mlx5_bond_1:1/path0 |
[1737025510.219908] [node13:2857558:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.220298] [node13:2857558:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1737025510.220302] [node13:2857558:0]   | perftest inter-node cfg#0 | active message by ucp_am_send*(multi) from host memory                                                                |
[1737025510.220305] [node13:2857558:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.220308] [node13:2857558:0]   |                    0..514 | short                                     | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1737025510.220310] [node13:2857558:0]   |                 515..4844 | zero-copy                                 | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1737025510.220313] [node13:2857558:0]   |                 4845..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_0:1/path0 and 50% on rc_mlx5/mlx5_bond_1:1/path0 |
[1737025510.220315] [node13:2857558:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.220495] [node13:2857558:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1737025510.220498] [node13:2857558:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* with reply flag from host memory                                                       |
[1737025510.220502] [node13:2857558:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.220504] [node13:2857558:0]   |                   0..2030 | short                                     | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1737025510.220507] [node13:2857558:0]   |                2031..8238 | copy-in                                   | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1737025510.220510] [node13:2857558:0]   |               8239..27188 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1737025510.220513] [node13:2857558:0]   |                27189..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_0:1/path0 and 50% on rc_mlx5/mlx5_bond_1:1/path0 |
[1737025510.220515] [node13:2857558:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.220661] [node13:2857558:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1737025510.220664] [node13:2857558:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* with reply flag(fast-completion) from host memory                                      |
[1737025510.220666] [node13:2857558:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.220670] [node13:2857558:0]   |                   0..2030 | short                                     | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1737025510.220672] [node13:2857558:0]   |                2031..8238 | copy-in                                   | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1737025510.220675] [node13:2857558:0]   |               8239..13104 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1737025510.220678] [node13:2857558:0]   |             13105..262143 | multi-frag zero-copy                      | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1737025510.220680] [node13:2857558:0]   |                 256K..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_0:1/path0 and 50% on rc_mlx5/mlx5_bond_1:1/path0 |
[1737025510.220683] [node13:2857558:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.220871] [node13:2857558:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1737025510.220875] [node13:2857558:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* with reply flag(multi) from host memory                                                |
[1737025510.220878] [node13:2857558:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.220881] [node13:2857558:0]   |                    0..514 | short                                     | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1737025510.220884] [node13:2857558:0]   |                 515..4844 | zero-copy                                 | rc_mlx5/mlx5_bond_0:1/path0                                               |
[1737025510.220887] [node13:2857558:0]   |                 4845..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_0:1/path0 and 50% on rc_mlx5/mlx5_bond_1:1/path0 |
[1737025510.220889] [node13:2857558:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.228624] [node13:2857558:a]          wireup.c:1214 UCX  ERROR   old: am_lane 0 wireup_msg_lane 4 cm_lane <none> keepalive_lane 4 reachable_mds 0x3
[1737025510.228640] [node13:2857558:a]          wireup.c:1237 UCX  ERROR   old: lane[0]:  2:rc_mlx5/mlx5_bond_0:1.0 md[0]  -> md[1]/ib/sysdev[0] seg 4294967295 rma_bw#1 am am_bw#0
[1737025510.228646] [node13:2857558:a]          wireup.c:1237 UCX  ERROR   old: lane[1]:  6:rc_mlx5/mlx5_bond_1:1.0 md[1]  -> md[0]/ib/sysdev[1] seg 4294967295 rma_bw#0
[1737025510.228650] [node13:2857558:a]          wireup.c:1237 UCX  ERROR   old: lane[2]:  2:rc_mlx5/mlx5_bond_0:1.1 md[0]  -> md[1]/ib/sysdev[0] seg 4294967295 rma_bw#3
[1737025510.228654] [node13:2857558:a]          wireup.c:1237 UCX  ERROR   old: lane[3]:  6:rc_mlx5/mlx5_bond_1:1.1 md[1]  -> md[0]/ib/sysdev[1] seg 4294967295 rma_bw#2
[1737025510.228658] [node13:2857558:a]          wireup.c:1237 UCX  ERROR   old: lane[4]:  3:ud_mlx5/mlx5_bond_0:1.0 md[0]  -> md[1]/ib/sysdev[0] seg 4294967295 keepalive wireup
[1737025510.228662] [node13:2857558:a]          wireup.c:1240 UCX  ERROR   old: err mode 1, flags 0x0
[1737025510.228667] [node13:2857558:a]          wireup.c:1214 UCX  ERROR   new: am_lane 0 wireup_msg_lane 3 cm_lane <none> keepalive_lane 0 reachable_mds 0x3
[1737025510.228671] [node13:2857558:a]          wireup.c:1237 UCX  ERROR   new: lane[0]:  2:rc_mlx5/mlx5_bond_0:1.0 md[0]  -> md[1]/ib/sysdev[0] seg 4294967295 rma_bw#1 am am_bw#0 keepalive
[1737025510.228676] [node13:2857558:a]          wireup.c:1237 UCX  ERROR   new: lane[1]:  6:rc_mlx5/mlx5_bond_1:1.0 md[1]  -> md[0]/ib/sysdev[1] seg 4294967295 rma_bw#0
[1737025510.228680] [node13:2857558:a]          wireup.c:1237 UCX  ERROR   new: lane[2]:  2:rc_mlx5/mlx5_bond_0:1.1 md[0]  -> md[1]/ib/sysdev[0] seg 4294967295 rma_bw#3
[1737025510.228684] [node13:2857558:a]          wireup.c:1237 UCX  ERROR   new: lane[3]:  6:rc_mlx5/mlx5_bond_1:1.1 md[1]  -> md[0]/ib/sysdev[1] seg 4294967295 rma_bw#2 wireup
[1737025510.228687] [node13:2857558:a]          wireup.c:1240 UCX  ERROR   new: err mode 1, flags 0x0
[node13:2857558:a:2857644]      wireup.c:1734 Fatal: endpoint reconfiguration not supported yet
==== backtrace (tid:2857644) ====
 0  /root/liufeng/ucx-1.18.0/install-release-mt/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f7fa5815e24]
 1  /root/liufeng/ucx-1.18.0/install-release-mt/lib/libucs.so.0(ucs_fatal_error_message+0xb2) [0x7f7fa5812fd2]
 2  /root/liufeng/ucx-1.18.0/install-release-mt/lib/libucs.so.0(+0x300a9) [0x7f7fa58130a9]
 3  /root/liufeng/ucx-1.18.0/install-release-mt/lib/libucp.so.0(ucp_wireup_init_lanes+0x1190) [0x7f7fa5943cc0]
 4  /root/liufeng/ucx-1.18.0/install-release-mt/lib/libucp.so.0(+0xb8e69) [0x7f7fa5943e69]
 5  /root/liufeng/ucx-1.18.0/install-release-mt/lib/libucp.so.0(+0xb9711) [0x7f7fa5944711]
 6  /root/liufeng/ucx-1.18.0/install-release-mt/lib/ucx/libuct_ib.so.0(uct_ud_ep_process_rx+0x22f) [0x7f7fa431a07f]
 7  /root/liufeng/ucx-1.18.0/install-release-mt/lib/ucx/libuct_ib_mlx5.so.0(+0x50284) [0x7f7fa3c14284]
 8  /root/liufeng/ucx-1.18.0/install-release-mt/lib/ucx/libuct_ib_mlx5.so.0(+0x4bd0b) [0x7f7fa3c0fd0b]
 9  /root/liufeng/ucx-1.18.0/install-release-mt/lib/libucs.so.0(+0x18874) [0x7f7fa57fb874]
10  /root/liufeng/ucx-1.18.0/install-release-mt/lib/libucs.so.0(ucs_async_dispatch_handlers+0x44) [0x7f7fa57fc554]
11  /root/liufeng/ucx-1.18.0/install-release-mt/lib/libucs.so.0(+0x1c096) [0x7f7fa57ff096]
12  /root/liufeng/ucx-1.18.0/install-release-mt/lib/libucs.so.0(ucs_event_set_wait+0xa1) [0x7f7fa58209d1]
13  /root/liufeng/ucx-1.18.0/install-release-mt/lib/libucs.so.0(+0x1c70e) [0x7f7fa57ff70e]
14  /lib64/libpthread.so.0(+0x81cf) [0x7f7fa4fad1cf]
15  /lib64/libc.so.6(clone+0x43) [0x7f7fa47d9d83]
=================================
[node13:2857558:a:2857644] Process frozen, press Enter to attach a debugger...

client:

[root@node12 ucx-1.18.0]# UCX_IB_ROCE_LOCAL_SUBNET=y UCX_PROTO_ENABLE=y UCX_TLS=rc  UCX_PROTO_INFO=y ./install-release-mt/bin/ucx_perftest 10.16.29.13 -t ucp_am_bw -s 1048576  -n 5000000 -e
[1737025510.109845] [node13:2866894:0]        perftest.c:800  UCX  WARN  CPU affinity is not set (bound to 96 cpus). Performance may be impacted.
+--------------+--------------+------------------------------+---------------------+-----------------------+
|              |              |       overhead (usec)        |   bandwidth (MB/s)  |  message rate (msg/s) |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
|    Stage     | # iterations | 50.0%ile | average | overall |  average |  overall |  average  |  overall  |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
[1737025510.214790] [node13:2866894:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1737025510.214801] [node13:2866894:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* from host memory                                                                       |
[1737025510.214806] [node13:2866894:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.214809] [node13:2866894:0]   |                   0..2038 | short                                     | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737025510.214810] [node13:2866894:0]   |                2039..8246 | copy-in                                   | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737025510.214813] [node13:2866894:0]   |               8247..27188 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737025510.214815] [node13:2866894:0]   |                27189..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_1:1/path0 and 50% on rc_mlx5/mlx5_bond_0:1/path0 |
[1737025510.214818] [node13:2866894:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.214985] [node13:2866894:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1737025510.214990] [node13:2866894:0]   | perftest inter-node cfg#0 | active message by ucp_am_send*(fast-completion) from host memory                                                      |
[1737025510.214993] [node13:2866894:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.214996] [node13:2866894:0]   |                   0..2038 | short                                     | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737025510.214999] [node13:2866894:0]   |                2039..8246 | copy-in                                   | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737025510.215002] [node13:2866894:0]   |               8247..13104 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737025510.215005] [node13:2866894:0]   |             13105..262143 | multi-frag zero-copy                      | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737025510.215007] [node13:2866894:0]   |                 256K..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_1:1/path0 and 50% on rc_mlx5/mlx5_bond_0:1/path0 |
[1737025510.215009] [node13:2866894:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.215385] [node13:2866894:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1737025510.215389] [node13:2866894:0]   | perftest inter-node cfg#0 | active message by ucp_am_send*(multi) from host memory                                                                |
[1737025510.215392] [node13:2866894:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.215395] [node13:2866894:0]   |                    0..514 | short                                     | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737025510.215399] [node13:2866894:0]   |                 515..4844 | zero-copy                                 | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737025510.215401] [node13:2866894:0]   |                 4845..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_1:1/path0 and 50% on rc_mlx5/mlx5_bond_0:1/path0 |
[1737025510.215404] [node13:2866894:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.215568] [node13:2866894:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1737025510.215572] [node13:2866894:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* with reply flag from host memory                                                       |
[1737025510.215574] [node13:2866894:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.215576] [node13:2866894:0]   |                   0..2030 | short                                     | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737025510.215578] [node13:2866894:0]   |                2031..8238 | copy-in                                   | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737025510.215579] [node13:2866894:0]   |               8239..27188 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737025510.215582] [node13:2866894:0]   |                27189..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_1:1/path0 and 50% on rc_mlx5/mlx5_bond_0:1/path0 |
[1737025510.215586] [node13:2866894:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.215729] [node13:2866894:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1737025510.215733] [node13:2866894:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* with reply flag(fast-completion) from host memory                                      |
[1737025510.215735] [node13:2866894:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.215739] [node13:2866894:0]   |                   0..2030 | short                                     | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737025510.215741] [node13:2866894:0]   |                2031..8238 | copy-in                                   | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737025510.215744] [node13:2866894:0]   |               8239..13104 | multi-frag copy-in                        | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737025510.215747] [node13:2866894:0]   |             13105..262143 | multi-frag zero-copy                      | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737025510.215750] [node13:2866894:0]   |                 256K..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_1:1/path0 and 50% on rc_mlx5/mlx5_bond_0:1/path0 |
[1737025510.215753] [node13:2866894:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.215928] [node13:2866894:0]   +---------------------------+-----------------------------------------------------------------------------------------------------------------------+
[1737025510.215931] [node13:2866894:0]   | perftest inter-node cfg#0 | active message by ucp_am_send* with reply flag(multi) from host memory                                                |
[1737025510.215934] [node13:2866894:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025510.215937] [node13:2866894:0]   |                    0..514 | short                                     | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737025510.215940] [node13:2866894:0]   |                 515..4844 | zero-copy                                 | rc_mlx5/mlx5_bond_1:1/path0                                               |
[1737025510.215943] [node13:2866894:0]   |                 4845..inf | (?) rendezvous zero-copy read from remote | 50% on rc_mlx5/mlx5_bond_1:1/path0 and 50% on rc_mlx5/mlx5_bond_0:1/path0 |
[1737025510.215947] [node13:2866894:0]   +---------------------------+-------------------------------------------+---------------------------------------------------------------------------+
[1737025544.016750] [node13:2866894:0]         libperf.c:1069 UCX  ERROR error handler called with status -80 (Endpoint timeout)

yosefe · 2025-01-16T13:07:01Z

@ivanallen can you pls try without the "-e" parameter?

ivanallen · 2025-01-17T05:26:00Z

@ivanallen can you pls try without the "-e" parameter?

@yosefe It works! Why?

ivanallen added the Bug label Jan 15, 2025

ivanallen closed this as completed Jan 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

multi-rails hang io #10420

multi-rails hang io #10420

ivanallen commented Jan 15, 2025 •

edited

Loading

ivanallen commented Jan 15, 2025

yosefe commented Jan 15, 2025

ivanallen commented Jan 16, 2025 •

edited

Loading

yosefe commented Jan 16, 2025

ivanallen commented Jan 16, 2025

yosefe commented Jan 16, 2025

ivanallen commented Jan 16, 2025 •

edited

Loading

yosefe commented Jan 16, 2025

ivanallen commented Jan 17, 2025

multi-rails hang io #10420

multi-rails hang io #10420

Comments

ivanallen commented Jan 15, 2025 • edited Loading

Describe the bug

Steps to Reproduce

Setup and versions

Additional information (depending on the issue)

ivanallen commented Jan 15, 2025

yosefe commented Jan 15, 2025

ivanallen commented Jan 16, 2025 • edited Loading

yosefe commented Jan 16, 2025

ivanallen commented Jan 16, 2025

yosefe commented Jan 16, 2025

ivanallen commented Jan 16, 2025 • edited Loading

yosefe commented Jan 16, 2025

ivanallen commented Jan 17, 2025

ivanallen commented Jan 15, 2025 •

edited

Loading

ivanallen commented Jan 16, 2025 •

edited

Loading

ivanallen commented Jan 16, 2025 •

edited

Loading