Fixed Grafana / Tempo communication #46
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
(I've now confirmed this works by running a cardano node and syncing Amaru) |
|
I could validate your changes via the datasource Could you detail a bit how you check things are fine with an amaru node? It's not clear to me how to use tempo from grafana. |
|
Grafana was no longer able to communicate with the Tempo backend after some config changes recently. Checking the Grafana logs, I found ``` 2024-12-27 04:52:35 logger=grafana-apiserver t=2024-12-27T09:52:35.291900754Z level=info msg="[core] [Channel #2 SubChannel #3]grpc: addrConn.createTransport failed to connect to {Addr: \"tempo:3200\", ServerName: \"tempo:3200\", }. Err: connection error: desc = \"error reading server preface: http2: frame too large\"" ``` After some googling, this seemed to be an indicator that the client (Grafana) was trying to connect to the backend (Tempo) with http2, but the backend only supporting http1.1. In particular, Tempo uses grpc for internal communication, but by default, exposes its grpc capabilities on the grpc port, 4317; However, we had Grafana connecting to the http port, 3200. One option would be to expose the grpc port and use that, but it seems this is common enough that there's a built in [flag](https://github.com/grafana/tempo/blob/main/example/docker-compose/shared/tempo.yaml#L1) for it. This causes tempo to expose its grpc API over http; in particular, this means that the http2 request to initiate the grpc connection now succeeds. I chose to go with this approach, since that's the way the [examples](https://github.com/grafana/tempo/blob/main/example/docker-compose/shared/tempo.yaml#L1) I saw were structured, and it's the first thing that worked. See also this github discussion for context: open-telemetry/opentelemetry-collector#7680
…mple. Prometheus is needed to collect metrics and serve them to grafana. While we don't have many metrics yet emitted directly by Amaru, we are able to automatically derive some from spans using the OpenTelemetry collector with the 'spanmetrics' collector/plugin.
|
Thanks 🫡 , seems to be fine. I knew it was some flag that got dropped when I moved around and pruned the configs. 
I also noticed that prometheus wasn't actually configured in the docker-compose file 🙈 ... so added it. 
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Grafana was no longer able to communicate with the Tempo backend after some config changes recently.
Checking the Grafana logs, I found
After some googling, this seemed to be an indicator that the client (Grafana) was trying to connect to the backend (Tempo) with http2, but the backend only supporting http1.1.
In particular, Tempo uses grpc for internal communication, but by default, exposes its grpc capabilities on the grpc port, 4317; However, we had Grafana connecting to the http port, 3200.
One option would be to expose the grpc port and use that, but it seems this is common enough that there's a built in flag for it.
This causes tempo to expose its grpc API over http; in particular, this means that the http2 request to initiate the grpc connection now succeeds. I chose to go with this approach, since that's the way the examples I saw were structured, and it's the first thing that worked.
See also this github discussion for context:
open-telemetry/opentelemetry-collector#7680
Note: I don't have a cardano node currently, so I wasn't able to test it with Amaru running; instead, I tested the tempo data source connection within Grafana itself, and I am assuming this is sufficient.
Before:
After:
