Publishing error when streaming Caliper data to LDMS

[cwschilly]

I am trying to configure Caliper to stream application data to LDMS, but I'm getting Error 107 publishing data when I run my app. I'm not sure if the issue is with my Caliper configuration, my LDMS configuration, or something else entirely. Any insight into the correct way to stream Caliper data to LDMS would be greatly appreciated.

Background

The app in question uses Kokkos, so there are no explicit Caliper annotations. I am simply exporting:

export KOKKOS_TOOLS_LIBS=/path/to/libcaliper.so
export CALI_CONFIG_FILE=/path/to/caliper.config

The caliper.config file is included below (Setup/Caliper configuration).

Setup

Caliper configuration

This caliper.config file is meant to capture function/region timings for MPI and Kokkos calls.

CALI_SERVICES_ENABLE=event,mpi,timestamp,trace,kokkostime,ldms
CALI_TIMER_SNAPSHOT_DURATION=false
CALI_TIMER_INCLUSIVE_DURATION=false
CALI_TIMER_OFFSET=true
CALI_MPI_MSG_TRACING=true
CALI_MPI_BLACKLIST=MPI_Comm_rank,MPI_Comm_size,MPI_Wtime

LDMS Configuration

This is in a file called cali.conf and is taken primarily from the LDMS docs. I'm using this to configure the LDMS daemon.

load name=hello_sampler
config name=hello_sampler producer=${HOSTNAME} instance=${HOSTNAME}/hello_sampler stream=caliper component_id=${COMPONENT_ID}
start name=hello_sampler interval=${SAMPLE_INTERVAL} offset=${SAMPLE_OFFSET}

# Store Caliper stream data to CSV
load name=stream_csv_store
config name=stream_csv_store path=./streams/store container=csv stream=caliper rolltype=3 rollover=500000

For now I'm just trying to send the data into a csv file. Later on, I would like to store it in a SOS database.

Starting LDMS Daemon

ldmsd -x sock:10003 -c cali.conf -l $LOGS/caliper.log &

Environment

I set all of these environment variables before running:

# Set LDMS connection environment variables (if not already set)
export CALIPER_LDMS_STREAM=${CALIPER_LDMS_STREAM:-"caliper"}
export CALIPER_LDMS_XPRT=${CALIPER_LDMS_XPRT:-"sock"}
export CALIPER_LDMS_HOST=${CALIPER_LDMS_HOST:-"localhost"}
export CALIPER_LDMS_PORT=${CALIPER_LDMS_PORT:-"10003"}
export CALIPER_LDMS_AUTH=${CALIPER_LDMS_AUTH:-"none"}
export CALIPER_LDMS_VERBOSE=${CALIPER_LDMS_VERBOSE:-1}

And then I run my application with mpiexec.

Output

During the run, I get some Caliper output followed by Error 107 publishing data on each MPI rank. Here is a snippet of the output:

{ "timestamp": 1761765904.588481, "jobid" : 0, "rank" : 7, "procid" : (null), "nodelist" : (null), "caliper-perf-data", "duration": 0.004718, "path": "Kokkos::View::initialization [CommMPI::pack_indicies_all] via memset"} 

{ "timestamp": 1761765909.000172, "jobid" : 0, "rank" : 7, "procid" : (null), "nodelist" : (null), "caliper-perf-data", "duration": 0.004515, "path": "Kokkos::View::initialization [CommMPI::pack_indicies_all] via memset"} 

{ "timestamp": 1761765909.000170, "jobid" : 0, "rank" : 1, "procid" : (null), "nodelist" : (null), "caliper-perf-data", "duration": 0.004464, "path": "Kokkos::Impl::ViewValueFunctor: View init/destroy fence"} 

{ "timestamp": 1761765909.000170, "jobid" : 0, "rank" : 1, "procid" : (null), "nodelist" : (null), "caliper-perf-data", "duration": 0.004464, "path": "Kokkos::View::initialization [CommMPI::pack_buffer]"} 

== CALIPER: (1): Error 107 publishing data.
== CALIPER: (1): Error 107 publishing data.
== CALIPER: (7): Error 107 publishing data.

Versions

Caliper v2.13.1
LDMS Versions:

LDMSD Version: 4.5.1-rc.1
LDMS Protocol Version: 4.2.0.0
LDMSD Plugin Interface Version: 3.2.2.0
git-SHA: 9e84000a5e6dd0942fc17e5be1309e744967ede2

Note: I get the same error with LDMSD 4.4.6.

[daboehme]

Hi @cwschilly, thanks for the report.

The error code comes directly out of an ldmsd_stream_publish call. I'm not sure what it means, I suppose something with the LDMS connection went wrong. The LDMS team might be better able to explain what is going on.

[cwschilly]

Thanks @daboehme you are right, there was a problem with my LDMS connection.

Can you help me understand a little more about how data is streamed from Caliper to LDMS? I am trying to get an MPI trace (containing MPI_Send, MPI_Allreduce, etc.) in addition to the region annotations via Kokkos tools. When I use the recorder service this seems to be possible, but when I use ldms it looks like all of the MPI regions are missing from the resulting csv.