Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

K8s Helm deploy seems to be buggy / broke #1816

Open
Ron-Norman opened this issue Dec 6, 2024 · 4 comments
Open

K8s Helm deploy seems to be buggy / broke #1816

Ron-Norman opened this issue Dec 6, 2024 · 4 comments
Labels
bug Something isn't working

Comments

@Ron-Norman
Copy link

Ron-Norman commented Dec 6, 2024 •
edited by julianocosta89
Loading

Bug Report

When deployed as helm chart, most grafana dashboard panels are showing "No Data" and when I expose Prometheus and view target help, most are broken. I have not done any customizations other that creating ingress for frontend and Prometheus. If I deploy on docker it all seems to work.

Which version of the demo you are using?
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
otel-demo default 2 2024-12-06 11:39:37.81596989 -0700 MST deployed opentelemetry-demo-0.33.7 1.12.0

Symptom

Capture1
Capture2
Capture3
Capture4

A clear and concise description of what the bug is.

When deployed as helm chart, most grafana dashboard panels are showing "No Data" and when I expose Prometheus and view target help, most are broken. I have not done any customizations other that creating ingress for frontend and Prometheus. If I deploy on docker it all seems to work. Also the dashboard service selector is not populated.

What do you expect to see?

Working the same as when deploy via docker.

Please describe the actual behavior experienced.

Same as above, see attach.

Could you provide the minimum required steps to resolve the issue you're seeing?

  1. deploy a default k3s - I trying on a multi-node set and single simple config.
  2. Deply otel demo with minimal customizations to expose the front-end and Prometheus.
  3. Access Grafana and Prometheus to see issue

We will close this issue if:

  • The steps you provided are complex.
  • If we can not reproduce the behavior you're reporting.

Additional Context

Please feel free to add any other context about the problem here.

Here is my custom values file

components:
  frontendProxy:
    ingress:
      enabled: true
      ## Added below to fix issue with not accessible via IC
      ingressClassName: traefik
      annotations: {}
      hosts:
        - host: otel-demo-lcl.home.lab
          paths:
            - path: /
              pathType: Prefix
              port: 8080
    service:
      type: LoadBalancer
    envOverrides:
      - name: PUBLIC_OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
        value: http://otel-demo.home.lab/otlp-http/v1/traces

prometheus:
server:
service:
type: LoadBalancer
loadBalancerIP: "10.10.230.111"

Kubectl output

ronanorm@rnws-linK|opentelemetry-demo$>> kubectl get all
NAME                                                   READY   STATUS    RESTARTS        AGE
pod/otel-demo-accountingservice-996db44fd-mdbnm        1/1     Running   0               23m
pod/otel-demo-adservice-6d9788b57d-kdj44               1/1     Running   0               23m
pod/otel-demo-cartservice-7cff65c78f-fcxkb             1/1     Running   0               23m
pod/otel-demo-checkoutservice-799bfd4698-2fkbp         1/1     Running   0               23m
pod/otel-demo-currencyservice-6b48b4d447-mvh2x         1/1     Running   0               23m
pod/otel-demo-emailservice-9c6cdbdd8-xjjg2             1/1     Running   0               23m
pod/otel-demo-flagd-dd64896b-zzjmv                     2/2     Running   0               23m
pod/otel-demo-frauddetectionservice-797598cc59-2dcvc   1/1     Running   0               23m
pod/otel-demo-frontend-f9d55f999-gj27b                 1/1     Running   1 (7m15s ago)   23m
pod/otel-demo-frontendproxy-587fd84d44-z9rnb           1/1     Running   0               23m
pod/otel-demo-grafana-75f688869c-4n45d                 1/1     Running   0               23m
pod/otel-demo-imageprovider-cb8c66b4c-ttwzh            1/1     Running   0               23m
pod/otel-demo-jaeger-5f44f99657-cbm29                  1/1     Running   0               23m
pod/otel-demo-kafka-6c4cfcb854-gjnl2                   1/1     Running   0               23m
pod/otel-demo-loadgenerator-5944bf844c-6mmpz           1/1     Running   0               23m
pod/otel-demo-opensearch-0                             1/1     Running   0               23m
pod/otel-demo-otelcol-7596b9ffd7-kh9tz                 1/1     Running   0               23m
pod/otel-demo-paymentservice-7b884d56dd-tklht          1/1     Running   0               23m
pod/otel-demo-productcatalogservice-5959cd684c-7644j   1/1     Running   0               23m
pod/otel-demo-prometheus-server-78869b8654-k2fv2       1/1     Running   0               23m
pod/otel-demo-quoteservice-bfb7cb5c6-j9j69             1/1     Running   0               23m
pod/otel-demo-recommendationservice-67f8fff94c-r2tfp   1/1     Running   0               23m
pod/otel-demo-shippingservice-75c98bd59f-4qzx2         1/1     Running   0               23m
pod/otel-demo-valkey-668fdfcffc-lpmch                  1/1     Running   0               23m

NAME                                      TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                     AGE
service/kubernetes                        ClusterIP      10.43.0.1       <none>        443/TCP                                                                     118m
service/otel-demo-adservice               ClusterIP      10.43.131.59    <none>        8080/TCP                                                                    24m
service/otel-demo-cartservice             ClusterIP      10.43.8.176     <none>        8080/TCP                                                                    24m
service/otel-demo-checkoutservice         ClusterIP      10.43.193.209   <none>        8080/TCP                                                                    24m
service/otel-demo-currencyservice         ClusterIP      10.43.118.119   <none>        8080/TCP                                                                    24m
service/otel-demo-emailservice            ClusterIP      10.43.130.50    <none>        8080/TCP                                                                    24m
service/otel-demo-flagd                   ClusterIP      10.43.239.96    <none>        8013/TCP,4000/TCP                                                           24m
service/otel-demo-frontend                ClusterIP      10.43.36.241    <none>        8080/TCP                                                                    24m
service/otel-demo-frontendproxy           LoadBalancer   10.43.73.122    10.10.10.25   8080:32699/TCP                                                              24m
service/otel-demo-grafana                 ClusterIP      10.43.71.136    <none>        80/TCP                                                                      24m
service/otel-demo-imageprovider           ClusterIP      10.43.58.74     <none>        8081/TCP                                                                    24m
service/otel-demo-jaeger-agent            ClusterIP      None            <none>        5775/UDP,5778/TCP,6831/UDP,6832/UDP                                         24m
service/otel-demo-jaeger-collector        ClusterIP      None            <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP,4317/TCP,4318/TCP                    24m
service/otel-demo-jaeger-query            ClusterIP      None            <none>        16686/TCP,16685/TCP                                                         24m
service/otel-demo-kafka                   ClusterIP      10.43.113.132   <none>        9092/TCP,9093/TCP                                                           24m
service/otel-demo-loadgenerator           ClusterIP      10.43.221.13    <none>        8089/TCP                                                                    24m
service/otel-demo-opensearch              ClusterIP      10.43.220.1     <none>        9200/TCP,9300/TCP,9600/TCP                                                  24m
service/otel-demo-opensearch-headless     ClusterIP      None            <none>        9200/TCP,9300/TCP,9600/TCP                                                  24m
service/otel-demo-otelcol                 ClusterIP      10.43.206.95    <none>        6831/UDP,14250/TCP,14268/TCP,8888/TCP,4317/TCP,4318/TCP,9464/TCP,9411/TCP   24m
service/otel-demo-paymentservice          ClusterIP      10.43.51.91     <none>        8080/TCP                                                                    24m
service/otel-demo-productcatalogservice   ClusterIP      10.43.31.239    <none>        8080/TCP                                                                    24m
service/otel-demo-prometheus-server       LoadBalancer   10.43.24.117    10.10.10.25   9090:32129/TCP                                                              24m
service/otel-demo-quoteservice            ClusterIP      10.43.234.227   <none>        8080/TCP                                                                    24m
service/otel-demo-recommendationservice   ClusterIP      10.43.4.64      <none>        8080/TCP                                                                    24m
service/otel-demo-shippingservice         ClusterIP      10.43.155.248   <none>        8080/TCP                                                                    24m
service/otel-demo-valkey                  ClusterIP      10.43.218.62    <none>        6379/TCP                                                                    24m

NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/otel-demo-accountingservice       1/1     1            1           23m
deployment.apps/otel-demo-adservice               1/1     1            1           23m
deployment.apps/otel-demo-cartservice             1/1     1            1           23m
deployment.apps/otel-demo-checkoutservice         1/1     1            1           23m
deployment.apps/otel-demo-currencyservice         1/1     1            1           23m
deployment.apps/otel-demo-emailservice            1/1     1            1           23m
deployment.apps/otel-demo-flagd                   1/1     1            1           23m
deployment.apps/otel-demo-frauddetectionservice   1/1     1            1           23m
deployment.apps/otel-demo-frontend                1/1     1            1           23m
deployment.apps/otel-demo-frontendproxy           1/1     1            1           23m
deployment.apps/otel-demo-grafana                 1/1     1            1           23m
deployment.apps/otel-demo-imageprovider           1/1     1            1           23m
deployment.apps/otel-demo-jaeger                  1/1     1            1           23m
deployment.apps/otel-demo-kafka                   1/1     1            1           23m
deployment.apps/otel-demo-loadgenerator           1/1     1            1           23m
deployment.apps/otel-demo-otelcol                 1/1     1            1           23m
deployment.apps/otel-demo-paymentservice          1/1     1            1           23m
deployment.apps/otel-demo-productcatalogservice   1/1     1            1           23m
deployment.apps/otel-demo-prometheus-server       1/1     1            1           23m
deployment.apps/otel-demo-quoteservice            1/1     1            1           23m
deployment.apps/otel-demo-recommendationservice   1/1     1            1           23m
deployment.apps/otel-demo-shippingservice         1/1     1            1           23m
deployment.apps/otel-demo-valkey                  1/1     1            1           23m

NAME                                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/otel-demo-accountingservice-996db44fd        1         1         1       23m
replicaset.apps/otel-demo-adservice-6d9788b57d               1         1         1       23m
replicaset.apps/otel-demo-cartservice-7cff65c78f             1         1         1       23m
replicaset.apps/otel-demo-checkoutservice-799bfd4698         1         1         1       23m
replicaset.apps/otel-demo-currencyservice-6b48b4d447         1         1         1       23m
replicaset.apps/otel-demo-emailservice-9c6cdbdd8             1         1         1       23m
replicaset.apps/otel-demo-flagd-dd64896b                     1         1         1       23m
replicaset.apps/otel-demo-frauddetectionservice-797598cc59   1         1         1       23m
replicaset.apps/otel-demo-frontend-f9d55f999                 1         1         1       23m
replicaset.apps/otel-demo-frontendproxy-587fd84d44           1         1         1       23m
replicaset.apps/otel-demo-grafana-75f688869c                 1         1         1       23m
replicaset.apps/otel-demo-imageprovider-cb8c66b4c            1         1         1       23m
replicaset.apps/otel-demo-jaeger-5f44f99657                  1         1         1       23m
replicaset.apps/otel-demo-kafka-6c4cfcb854                   1         1         1       23m
replicaset.apps/otel-demo-loadgenerator-5944bf844c           1         1         1       23m
replicaset.apps/otel-demo-otelcol-7596b9ffd7                 1         1         1       23m
replicaset.apps/otel-demo-paymentservice-7b884d56dd          1         1         1       23m
replicaset.apps/otel-demo-productcatalogservice-5959cd684c   1         1         1       23m
replicaset.apps/otel-demo-prometheus-server-78869b8654       1         1         1       23m
replicaset.apps/otel-demo-quoteservice-bfb7cb5c6             1         1         1       23m
replicaset.apps/otel-demo-recommendationservice-67f8fff94c   1         1         1       23m
replicaset.apps/otel-demo-shippingservice-75c98bd59f         1         1         1       23m
replicaset.apps/otel-demo-valkey-668fdfcffc                  1         1         1       23m

NAME                                    READY   AGE
statefulset.apps/otel-demo-opensearch   1/1     23m
ronanorm@rnws-linK|opentelemetry-demo$>> kubectl get ingress
NAME                      CLASS     HOSTS                    ADDRESS       PORTS   AGE
otel-demo-frontendproxy   traefik   otel-demo-lcl.home.lab   10.10.10.25   80      24m
@Ron-Norman Ron-Norman added the bug Something isn't working label Dec 6, 2024
@julianocosta89
Copy link
Member

Hello @Ron-Norman 👋🏽

Thanks for reporting this.

We have updated the helm chart last week, are you using the latest version?

@puckpuck
Copy link
Contributor

I wonder if setting the service type to LoadBalancer on frontendProxy is the culprit here. All data including OTLP telemetry goes through it.

Can you try to deploy without the ingress route and see if it works?

@puckpuck puckpuck mentioned this issue Dec 12, 2024
Merged
@puckpuck
Copy link
Contributor

I've been able to replicate this on my end. It was due to this PR, which upgraded the Prometheus version to 3.x and made a few breaking changes with the OTLP receiver.

I have a fix tested and ready to go.

@Ron-Norman
Copy link
Author

Ron-Norman commented Dec 29, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@puckpuck @julianocosta89 @Ron-Norman