[Feature Request] Parsing response content of Non HTTP 200 response

[beirtipol]

The Spring boot /acuator/health endpoint will return a 503 error code if any of the components it is monitoring report a status of DOWN. The response body still contains valid JSON, but infinity doesn't attempt to parse it, it just says 'No data'.

This should still work, regardless of the response code.

Here's some sample JSON

{
    "status": "UP",
    "components": {
        "clientConfigServer": {
            "status": "UP",
            "details": {
                "propertySources": [
                    "configserver:[email protected]:MyCompany/myapp-config.git/thisapp/thisapp-uat.yaml",
                    "configClient"
                ]
            }
        },
        "diskSpace": {
            "status": "UP",
            "details": {
                "total": 107317563392,
                "free": 24000552960,
                "threshold": 10485760,
                "path": "/application/.",
                "exists": true
            }
        },
        "someCustomCheck": {
            "status": "DOWN",
            "details": {
                "everything is": "awful"
            }
        },
        "livenessState": {
            "status": "UP"
        },
        "ping": {
            "status": "UP"
        },
        "readinessState": {
            "status": "UP"
        },
        "refreshScope": {
            "status": "UP"
        }
    },
    "groups": [
        "liveness",
        "readiness"
    ]
}

[yesoreyeram]

This is the expected behaviour of the plugin. How do you differentiate the 503 by the API error vs 503 of the underlying application health?

I would consider this as a feature request but not a bug.

[yesoreyeram]

actually this piece of code considers anything greater than HTTP 400 as error.

grafana-infinity-datasource/pkg/infinity/client.go

Lines 218 to 220 in 6a6fa3a

	if res.StatusCode >= http.StatusBadRequest {
	return nil, res.StatusCode, duration, errors.New(res.Status)
	}

We can accept something like ignoreHttpStatusCode: true in query then ignore http status codes in such cases

[beirtipol]

I guess I think of the plugin as something that is purely for fetching data rather than error codes. I appreciate it might not be a typical use case, but the data contained in the error response is what's useful. Even with something like a 404, it is usually returned by a web server to indicate the page doesn't exist, but it is also totally valid to indicate that the resource you queried for doesn't exist, and the response might tell you more useful info. In an ideal world, you would also be able to append the error code to the response object for something like colour coding the widget.

…

On Mon, 10 Jun 2024, 18:18 Sriram, ***@***.***> wrote: This is the expected behaviour of the plugin. How do you differentiate the 503 by the API error vs 503 of the underlying application health? I would consider this as a feature request but not a bug. — Reply to this email directly, view it on GitHub <#874 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJQO3DYJQVGHF3NWY3KCKLZGXNVLAVCNFSM6AAAAABJCQNDEOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJYHEYDMMBZGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Houlistonm]

We have the same problem. Sadly, we're trying to use the same endpoint /actuator/health for two different purposes and the clients have different requirements.
The /actuator/health is scraped by:

1. cloudfoundry foundation and the http response code is used to determine if the instance is health or not.
2. grafana using the infinity data source to parse the payload to determine if the instance is healthy or not. unfortunately, the payload never gets read because of the 503 response when the i instance is unhealthy.

Spring boot does let us override the http response code (via configuration) to allow us to read the payload during an unhealthy state, but then the cloudfoundry will not replace the failed instance since the response code is not '503'

Is there a mechanism in the infinty data source to set a list of allowed http response codes via configuration (200, 503) so we can read the payload and take send alerts.

To be completely transparent, spring boot allows for the addition of code to expose the /actuator/health endpoint as a metric on the /actuator/prometheus endpoint for alerting, but that's just more code that has to be maintained by our development teams.