OTEL Collector: allow plain json payloads

[rauanmayemir]

I'm using vector.dev for collecting kubernetes logs and attempted to route cnpg cluster logs to pganalyze collector.

However, vector doesn't support proper OTLP sink, so initially I thought I could send plain otlp 'flavored' json as http payload, but pganalyze collector refused after failing to unmarshal protobuf.

[lfittl]

@rauanmayemir Thanks for the report!

Could you share your vector configuration you're using? (with any credentials/identifying information removed) -- that makes it easier to verify on our end what needs to be done.

Also if you could share the specific error message from the collector that would be helpful.

[rauanmayemir]

@lfittl I prepared a vector test repro, will send pganalyze request/response later. https://gist.github.com/rauanmayemir/d1a22a6e71b79c45c4854fa6db8a01df

vector test ./test_cnpg.toml

Warning log will show the output json payload that goes into pganalyze.

[rauanmayemir]

No matter what I try to send to the /v1/logs endpoint, this is the error collector returning to me:

E Could not unmarshal otel body

The handler is trying to parse the logs data from binary:

collector/input/system/selfhosted/otel_handler.go

Line 119 in 9b23d29

if err := proto.Unmarshal(b, logsData); err != nil {

I can create a json of the logsData structure and wrap my log entry into 'resource_logs->scope_logs->log_records', but handler has to understand that payload is not binary data.

[rauanmayemir]

Any update on this?

[rauanmayemir]

Here's the updated parsing logic that makes sure data structure is valid: https://gist.github.com/rauanmayemir/d1a22a6e71b79c45c4854fa6db8a01df

I added an extra step that sends logs from vector to opentelemetry collector first (via HTTP/json) and then from opentelemetry to pganalyze. It still doesn't work. I wish someone could confirm otel log collection has actually worked for them.

Here's the relevant otlp collector config part:

receivers:
  otlp/pganalyze:
    protocols:
      http:
        endpoint: ${env:MY_POD_IP}:14318
exporters:
  debug:
    verbosity: detailed
  otlphttp/pganalyze:
    endpoint: http://mydb-pganalyze-collector.myns.svc.cluster.local:4318
    tls:
      insecure: true
service:
  pipelines:
    logs/pganalyze:
      receivers: [ otlp/pganalyze ]
      processors: [  ]
      exporters: [ debug, otlphttp/pganalyze ]

Here's debug exporter log:

2025-03-01T13:06:07.349Z info Logs {"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 1}
2025-03-01T13:06:07.349Z info ResourceLog #0
Resource SchemaURL:
Resource attributes:
 -> source_type: Str()
 -> service.name: Str(mydb)
 -> host.hostname: Str(mydb-3)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope
LogRecord #0
ObservedTimestamp: 2025-03-01 13:05:56.97167799 +0000 UTC
Timestamp: 2025-03-01 13:05:56.971978966 +0000 UTC
SeverityText: INFO
SeverityNumber: Unspecified(0)
Body: Map({"error_severity":"info","logger":"postgres","record":{"backend_type":"autovacuum worker","error_severity":"LOG","log_time":"2025-03-01 13:05:56.971 UTC","message":"automatic vacuum of table \"mydb.pg_catalog.pg_statistic\": index scans: 0\npages: 0 removed, 234 remain, 201 scanned (85.90% of total)\ntuples: 0 removed, 11952 remain, 11185 are dead but not yet removable\nremovable cutoff: 1421524, which was 319850 XIDs old when operation ended\nfrozen: 0 pages from table (0.00% of total) had 0 tuples frozen\nindex scan not needed: 0 pages from table (0.00% of total) had 0 dead item identifiers removed\navg read rate: 0.000 MB/s, avg write rate: 0.000 MB/s\nbuffer usage: 423 hits, 0 misses, 0 dirtied\nWAL usage: 0 records, 0 full page images, 0 bytes\nsystem usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s","process_id":"3730478","query_id":"0","session_id":"67c30634.38ec2e","session_line_num":"1","session_start_time":"2025-03-01 13:05:56 UTC","sql_state_code":"00000","transaction_id":"0","virtual_transaction_id":"48/867424"}})
Trace ID:
Span ID:
Flags: 0
 {"kind": "exporter", "data_type": "logs", "name": "debug"}

pganalyze collector is still returning E Could not unmarshal otel body.