Often getting "no data" in response to an instant query

[lwithers]

Describe the bug

When I am trying to display some fairly static metric, and/or some information label, I will often use a Grafana "table" or "stat" panel and set the Type to instant (on the assumption that it's going to generate a fair bit less load to just load the value at one point in time than whatever time range the dashboard would have selected).

However, when I do this, I have found that I'm often getting intermittent results. On some refreshes, the value loads fine; on others, I get "No data".

Here is an example of the query inspector returning "No data" for such a query:

{
  "request": {
    "url": "api/ds/query?ds_type=victoriametrics-metrics-datasource&requestId=SQR283",
    "method": "POST",
    "data": {
      "queries": [
        {
          "editorMode": "code",
          "exemplar": false,
          "expr": "nvidia_device_count{job=\"nvidia-exporter\",instance=\"gpu0.rnd.int.ppd-ld5.gbr.internal.yoti.com\"}",
          "format": "table",
          "instant": true,
          "legendFormat": "__auto",
          "range": false,
          "refId": "A",
          "datasource": {
            "type": "victoriametrics-metrics-datasource",
            "uid": "afhs9qne1bd34c"
          },
          "queryType": "timeSeriesQuery",
          "requestId": "29A",
          "utcOffsetSec": 3600,
          "interval": "",
          "datasourceId": 3,
          "intervalMs": 15000,
          "maxDataPoints": 1250
        }
      ],
      "from": "1775133253235",
      "to": "1775135053235"
    },
    "hideFromInspector": false
  },
  "response": {
    "results": {
      "A": {
        "status": 200,
        "frames": [],
        "refId": "A"
      }
    }
  }
}

To Reproduce

Create a "Stat" or "Table" panel with a query.

Set Type to instant.

Refresh over the course of several minutes and observe inconsistent results.

Here is an example panel:

{
  "id": 29,
  "type": "stat",
  "title": "Number of cards",
  "gridPos": {
    "x": 0,
    "y": 0,
    "h": 4,
    "w": 3
  },
  "fieldConfig": {
    "defaults": {
      "mappings": [],
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {
            "color": "text",
            "value": null
          }
        ]
      },
      "color": {
        "mode": "thresholds"
      }
    },
    "overrides": [
      {
        "matcher": {
          "id": "byName",
          "options": "Time"
        },
        "properties": []
      },
      {
        "matcher": {
          "id": "byName",
          "options": "uuid"
        },
        "properties": [
          {
            "id": "displayName",
            "value": "Device type"
          }
        ]
      },
      {
        "matcher": {
          "id": "byName",
          "options": "Value"
        },
        "properties": [
          {
            "id": "displayName",
            "value": "Count"
          }
        ]
      }
    ]
  },
  "pluginVersion": "12.4.2",
  "targets": [
    {
      "editorMode": "code",
      "exemplar": false,
      "expr": "nvidia_device_count{job=\"${job}\",instance=\"${instance}\"}",
      "format": "table",
      "instant": true,
      "legendFormat": "__auto",
      "range": false,
      "refId": "A"
    }
  ],
  "datasource": {
    "type": "victoriametrics-metrics-datasource",
    "uid": "afhs9qne1bd34c"
  },
  "options": {
    "reduceOptions": {
      "values": false,
      "calcs": [
        "lastNotNull"
      ],
      "fields": ""
    },
    "orientation": "auto",
    "textMode": "auto",
    "wideLayout": true,
    "colorMode": "value",
    "graphMode": "area",
    "justifyMode": "auto",
    "showPercentChange": false,
    "percentChangeColorMode": "standard"
  }
}

Versions of VictoriaMetrics datasource and VictoriaMetrics backend

• Grafana: v12.4.2
• datasource plugin: v0.23.3
• backend: 1.139.0
• vmauth: 1.139.0

Link to dashboard in VictoriaMetrics grafana playground

I created this dashboard: https://docs.victoriametrics.com/victoriametrics/sd_configs/#static_configs

but I was unable to reproduce the issue there.

Please provide dashboard JSON if it is possible

No response

Additional information

No response

[tiny-pangolin]

what is the scrape interval for the data you are trying to visualize, and the step in the dashboard panel? if the step is shorter than the scrape interval then the instant query used in those panels will show no data. try setting the step to at least 2x the scrape interval in the query.

[arturminchukov]

@lwithers, thanks for the bug report!
This issue can be related to this one -VictoriaMetrics/VictoriaMetrics#10477.

Could you please try to make a direct request to the VM backend to figure out that the backend returns some data, as described in the comment VictoriaMetrics/VictoriaMetrics#10477 (comment)

Also, pay attention to the comment above.

[lwithers]

Hi, thank you for the prompt replies.

I am using vmagent with a scrape_interval of 30s:

# defaults for all scrapes (can be overridden by specific scrape config files)
global:
  scrape_interval: 30s
  scrape_timeout: 28s

I checked in Grafana and I had not applied any minimum interval / step, so I changed each visualisation to have a 30s minimum interval. Since doing that, I have not seen any missing data.

---

If I query the data source directly, with a step af 15s then I see intermittent results:

$ curl -s -k -H 'Authorization: Bearer «redacted»' 'https://vmapi.nonprod-metrics.infra.yoti.com/api/v1/query?query=nvidia_device_count%7Binstance%3D%22gpu0.rnd.int.ppd-ld5.gbr.internal.yoti.com%22%7D&step=15s&time=' | json_pp

{
   "data" : {
      "result" : [],
      "resultType" : "vector"
   },
   "stats" : {
      "executionTimeMsec" : 7,
      "seriesFetched" : "1"
   },
   "status" : "success"
}

or:

{
   "data" : {
      "result" : [
         {
            "metric" : {
               "__name__" : "nvidia_device_count",
               "cluster_loc" : "ppd1",
               "instance" : "gpu0.rnd.int.ppd-ld5.gbr.internal.yoti.com",
               "job" : "nvidia-exporter"
            },
            "value" : [
               1775567985,
               "2"
            ]
         }
      ],
      "resultType" : "vector"
   },
   "stats" : {
      "executionTimeMsec" : 1,
      "seriesFetched" : "1"
   },
   "status" : "success"
}

If I repeat the query with a step of 30s, then I always get a non-empty result.

---

So, it seems the solution is simple, but just something that I was not aware of. Maybe it is worth mentioning in an FAQ or similar? (Or perhaps I have missed it?).

I'll note that I have a mix of two sampling intervals in my data: 15s for customer-facing applications, and 30s for backend infrastructure. The motivation for this is stretching the storage a bit further while focussing on the more sensitive data. For this reason I think it would not be suitable to set the Grafana datasource's scrape interval to 30s.

[tiny-pangolin]

I think we should mention the following in the Grafana integration docs

1. it generally recommeded to have a consistent scrape interval across your
2. a reference to the robust preception article explain why it is genrally a good idea to use 1 scrape interval