Does Skywalking support collecting trace date from LLM applications

[foxyriver]

Is there any practice of collecting LLM application trace date in Skywalking? I can see the LLM model calls, chain calls, and embedding calls。Even collecting metrics of token response time

[foxyriver]

@wallezhang @augustowebd @dio @Videl

[wu-sheng]

We don't have a direct solution out-of-the-box for that, AFAIK. But we support OTEL metrics/traces/logs from any source.

[foxyriver]

Our other microservices collect trace data through the skywalking agent. If we enable the SW-OTEL-RECEIVENR_ENABLED_HANDLERS configuration to collect OTEL traces, will it conflict with the data collected by the skywalking agent?

[wu-sheng]

It would not conflict, but the traces do not continue between these two ecosystems.
And OTEL traces are same as Zipkin traces, they are only showing up in zipkin lens UI(bundled or zipkin released one).
Ref to https://skywalking.apache.org/docs/main/latest/en/setup/backend/otlp-trace/

[wu-sheng]

Also, you would not have topology and metrics from them, because from their design perspective, these traces are enhanced logs, you should set up sampling rate, rather than 100% sampling.

[foxyriver]

This is the problem, we hope that LLM applications and non LLM applications can be seen in the same call chain, because we cannot clearly separate LLM applications from non LLM applications and make them perfectly in two systems, as there will be calls between them

[wu-sheng]

Then, you must build the plugins for the LLM clients.