Horizon Events API sends textual layer values to OAP, but records-all.layer is mapped as integer in OpenSearch

[jibarra88]

Question

Is this a known compatibility issue between Horizon UI 1.0.0 and OAP 11.x with OpenSearch storage?

Should Horizon convert layer keys like KAFKA, VIRTUAL_DATABASE, VIRTUAL_MQ to numeric layer IDs before calling OAP Events APIs?

Description

We are seeing an issue in Apache SkyWalking Horizon when opening the Events view for non-standard/virtual layers such as:

• KAFKA
• VIRTUAL_DATABASE
• VIRTUAL_MQ
• VIRTUAL_CACHE

The main dashboards, services, instances and endpoints pages work correctly. However, when the Events panel is opened, Horizon shows:

OAP is unreachable. Exception while fetching data (/data)

The OAP itself is reachable and GraphQL responds correctly.

Environment

• Apache SkyWalking OAP: 11.0.0-6f1fd78
• Apache SkyWalking Horizon UI: 1.0.0
• Storage: OpenSearch
• Storage namespace: skywalking-11
• Deployment: 2 OAP nodes + Horizon UI + OpenSearch cluster

OAP health check

The OAP GraphQL endpoint is healthy:

curl -s http://<oap-host>:12800/graphql \OAP is unreachable.
Exception while fetching data (/data):
java.util.concurrent.ExecutionException:
java.lang.RuntimeException:
{
  "error": {
    "root_cause": [
      {
        "type": "query_shard_exception",
        "reason": "failed to create query: For input string: \"KAFKA\"",
        "index": "skywalking-11_records-all-YYYYMMDD"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "caused_by": {
      "type": "number_format_exception",
      "reason": "For input string: \"KAFKA\""
    },
    "status": 400
  }
}


  -H 'Content-Type: application/json' \
  -d '{"query":"query { version }"}'
  
  Error message
  

### OpenSearch mapping
The mapping of records-all shows that layer is stored as an integer:
"layer": {
  "type": "integer",
  "doc_values": false
}
  
### Additional notes

Deleting skywalking-11_records-all-* does not permanently solve the issue. OAP recreates the indices with the same mapping:

"layer": {
  "type": "integer"
}

The issue comes back as soon as the Events view queries the recreated indices.

[sueun-dev]

See #14086 and PR #14087, merged September 17. That patch converts layer names in OAP's Elasticsearch query code, not Horizon. Your listed 6f1fd78 source still has the original string filter.

I ran commit 029571c7's unmodified Layer class in an isolated Java 21 test: KAFKA, VIRTUAL_DATABASE, VIRTUAL_MQ and VIRTUAL_CACHE converted to numeric IDs. This checks the conversion, not the query builder or OpenSearch. Compare that upstream change with your OAP build before changing Horizon's payload.

[jibarra88]

Thanks for the clarification and for pointing us to #14086 and PR #14087.

We confirmed that our OAP build 11.0.0-6f1fd78 does not include this fix, and the behavior we are seeing matches the known issue exactly.

I also noticed that the fix is already assigned to the 11.1.0 milestone, so we will plan our upgrade around that version.

Do you have any tentative release date or estimated release window for SkyWalking 11.1.0?

By the way, congratulations to the SkyWalking team. Horizon and the new 11.x experience look incredibly good. The product has improved a lot, and we are really enjoying the new UI and observability capabilities.

Thanks again for your support.