Data charts improvements

[jarongrigat]

It turns out people mostly use the internal doberview charts in the sensor detail modal instead of going to Grafana, so let's make them as good as possible.

• When you zoom in the sensor details plot, the same data points remain available. That means if you zoom in a lot the resolution is bad. Grafana, in contrast, reloads the data when you zoom (recalculating the binning). Would something like that be possible here? (copied from Improved zooming in sensor details plot #36)
  Another possible idea here is to load all the data with minimal binning (loading 100k data points from influxDB takes 0.5s or so so there is no need to be so picky here) and then bin graphically instead of on the data level.
• The show/hide outliers button should remove outliers from the dataset instead of changing the limits. It looks a little bit weird otherwise (see plot below).
• I really dislike the 2D-zoom - I think it's very unintuitive. I would consider removing it since it shouldn't be necessary once you remove outliers.
• Consider live updating plot. Refresh every 5s or so. Check performance.
• Consider switching from Highcharts to something else. Worth looking into canvasJS and charts.js
• Consider plotting alarm thresholds from data instead of just a line for current value. I don't remember if we had the discussion on storing alarm thresholds in influx before and came to the conclusion not to do it, but I think we can talk about it again.

@TiffanyLuce let me know what you think and if you want to have a look into this. If not, it's also fine - I don't think it's urgent or anything.

[TiffanyLuce]

Nice thing to gather all plots related issues there!

For the first issue, I already looked into it and was waiting for the xenon run to end before testing stuff. I once tried to plot 4 weeks data with a 1s binning and we got a storm of alarms because nothing else could be done at the same time... I am now trying to be more considerate.

I agree for the outliers looking weird and can remove them but we then loose the info on when they happened graphically, do we care? Also, some values are considered 'outliers' by my method while not actually being outliers, as for the thermosiphon top block (attached) or GaNi pressure for example. Removing them would remove valuable information. Maybe then also change the method of detecting outliers I don't know.

I agree with xy zoom and would like to only put a x axis zoom, more fitted with the outliers removing thing. But I don't know other people opinion

I don't know about live updating plots, the refresh button is already there to refresh and I feel like the plot is here to show the tendency and one can look at the overview or system page if they want to observe the value updating. Also, live refreshing might sabotage you if you're trying to focus your mouse on a point to get the value or something.

[jarongrigat]

Yes, let's wait for pancake downtime to test this :) Maybe we can also change some influx setting to avoid one user of blocking all other reads/writes.
Let's tag @slindemann @MuellerJulie @adambrown1 to see if they have some wishes and ideas.

[leeoniya]

Consider switching from Highcharts to something else. Worth looking into canvasJS and charts.js

shameless plug for the underlying lib that's used in Grafana :) https://github.com/leeoniya/uPlot

[adambrown1]

Consider live updating plot. Refresh every 5s or so. Check performance.

I like this idea, if you can find a way to do it. I think adding new data points is easyish from looking briefly into it before. And they get pulled every 5 seconds from the database anyway...

Consider plotting alarm thresholds from data instead of just a line for current value. I don't remember if we had the discussion on storing alarm thresholds in influx before and came to the conclusion not to do it, but I think we can talk about it again.

Sounds nice but probably not high priority

I really dislike the 2D-zoom - I think it's very unintuitive. I would consider removing it since it shouldn't be necessary once you remove outliers.

I must admit I really like the 2D zoom, for example when using a plot like Tiffany's example above

The show/hide outliers button should remove outliers from the dataset instead of changing the limits. It looks a little bit weird otherwise (see plot below).

I see both arguments here, but am more interested in functionality than good looks and so lean towards the current implementation. For presentation/publication quality plots it is clear we need to get the data offline and use a more flexible plotting library like matplotlib.

Consider switching from Highcharts to something else. Worth looking into canvasJS and charts.js

I guess we need to understand what we want from the plots that we don't currently get to make this decision

One more:
It would be really nice if we were able to add additional sensors to the plot, and use custom time ranges excluding the current time. Then I doubt I would ever go to grafana any more. Maybe even as a separate tool on its own page? But again, not something to put too much manpower on if it is a large effort

[TiffanyLuce]

Would it also be nice to be able to choose to view data as points rather than lines? I think it makes flickery data more readable (like the levelmeters)
It would at the same time 'solve' the outliers looking weird when you don't show them since they would be no lines getting out of the plot window to link to a far away point.

[adambrown1]

Would it also be nice to be able to choose to view data as points rather than lines? I think it makes flickery data more readable (like the levelmeters) It would at the same time 'solve' the outliers looking weird when you don't show them since they would be no lines getting out of the plot window to link to a far away point.

I wouldn't say this should necessarily be top priority but in principle sounds good