Format of data for large number of points to be a list instead of dict

[abbas-mohd56]

I'm trying to plot a LineSeries with 4000+ points. When I try to render the chart, It fails on the front end and shows me this link in the console https://assets.highcharts.com/errors/12/.

It fails on the front end because the number of points exceeds the default turboThreshold value which is 1000. I can disable turbo mode by setting it to 0 and it works just fine, However, if the data returned could be in the format which could be supported by boost mode, it'd be great!

To explain this with an example, imagine this DataFrame saved in variable df (Using 3 points instead of 4000 for simplicity):

timestamp	value
2023-08-03	19517.0
2023-08-04	21262.0
2023-08-05	18535.0

If I create a LineSeries instance using the following code:

from highcharts_core.options.series.area import LineSeries

series = LineSeries.from_pandas(df, {"x": "timestamp", "y": "value"})
print(series.to_json())

This gives me the following ouput:

{"data": [{"x": 1691020800000.0, "y": 19517.0}, {"x": 1691107200000.0, "y": 21262.0}, {"x": 1691193600000.0, "y": 18535.0}], "type": "line"}

However, for it to work in turbo mode, I need the data to be in the following format

{"data": [[1691020800000.0, 19517.0], [1691107200000.0, 21262.0], [1691193600000.0, 18535.0]], "type": "line"}

Is it somehow possible to generate this output?
Regardless of the turbo mode, this format would be ideal to send large datasets through an API as the size generated would be less compared to the list of dict format.

[hcpchris]

Hi @abbas-mohd56 : Yes, this should happen automatically as of v.1.3.0 following the implementation of #77 when serializing to a JS literal object. This means that when you call series.to_js_literal() or chart.to_js_literal() you should get a primitive array (not the object form).

The basic logic is as follows:

1. Highcharts for Python checks to see whether you have configured any other data point properties than those which Highcharts (JS) supports as a primitive array (the non-object representation).
2. If you have not configured any other data point properties, then that means that Highcharts for Python can return a simple primitive array rather than a list of objects.
3. If you have configured other data point properties, then that prevents Highcharts for Python from generating a primitive array and instead it gets serialized to a list of objects (which is more verbose, harder to read, less performant, etc.).

However, this logic is not applied when serializing to JSON (using the .to_json() method), since there we are producing a more robust / "complete" / "unambiguous" representation of the Highcharts for Python object (not the Highcharts (JS) object,
which is best represented as a JS literal object to take advantage of JS callback / formatter functions, etc.).

So given all of the above, I would recommend that you use series.to_js_literal() and use the JS literal form of the object, rather than series.to_json(). That should give you the kind of output that you are looking for as described above. If for some reason you cannot do this in your use case, I'd be curious to understand the reason. We can add the same functionality as in #77 to the .to_json() method - but that would be a slight shift in our thinking about how the JSON representation gets used. So if that's important for you, it would be very helpful to understand your reasoning.

[hcpchris]

Oh, and another alternative is to reset the turbo threshold, which you can do by configuring the series' turboThreshold property. In your code you could do this by setting: series.turbo_threshold = 5000 (for example), and turbo mode will still be applied even if you get a list of 4000 objects.

[abbas-mohd56]

Let me explain to you you how I've integrated Highcharts for Python in my project.
I'm using Django as my web framework. The loading of charts and populating it with series has been split into 2 different requests. I've simplified the code wherever possible to explain how I'm using it, but let me know if you don't understand anything from here

Step 1: Rendering the chart

I have a shared_options.js with the following options:

{
  chart: {
    zooming: {
      type: "x"
    },
  },
  tooltip: {
    shared: true
  },
  xAxis: {
    type: "datetime",
  },
  yAxis: {
    title: {
      text: " "
    }
  }
}

I configure my chart in views.py and add it to the template context

from django.views.generic.base import TemplateView
from django.utils.safestring import mark_safe
from highcharts_core.highcharts import SharedOptions
from highcharts_core.chart import Chart


class DashboardView(TemplateView):
    template_name = "dashboard.html"

    def get_context_data(self, **kwargs):
        # Get context dictionary
        context = super().get_context_data(**kwargs)
        # Configure Highcharts Instances
        shared_options = SharedOptions.from_js_literal("<path>/shared_options.js")
        chart = Chart(container="chart_id", variable_name="chart")
        # Add Highchart instances to template context
        context["shared_options"] = mark_safe(shared_options.to_js_literal())
        context["chart"] = mark_safe(chart.to_js_literal())
        return context

And finally I have the following snippet of code on my template to initialize the chart

<div id="chart_id"></div>
<script>
  {{ shared_options }}
  {{ chart }}
</script>

So far, I do not face any issue with the code

Step 2: Preparing the series for the chart

The Rendered view makes an API call to the backend and prepares a Series object based on Users input. In this case, it'd be start and end date.
The LineSeries object is generated by the following view

from django.http import JsonResponse
from django.views import View
from highcharts_core.options.series.area import LineSeries
from .utils import get_df


class DashboardAPIView(View):
    def get(self, request, *args, **kwargs):
        start_date = request.GET.get("startDate")
        end_date = request.GET.get("endDate")
        df = get_df(start_date, end_date)
        # df is a simple DataFrame with 2 columns: timestamp and value
        # timestamp         value
        # 2023-08-01        10
        # 2023-08-02        15
        # df could have a lot of rows depending on the start and end dates

        # Generating LineSeries
        series = LineSeries.from_pandas(
            df, property_map={"x": "timestamp", "y": "value"}
        )
        # Here I can either return series.to_json() or series.to_js_literal()
        # because both return strings, but I can only parse the string
        # generated from to_json() using JSON.parse on my template. Is there a
        # way to parse JS literal? that'd solve my problem.
        return JsonResponse({
            "seriesData":  series.to_json()
        })

This is the snippet of code I use on my template to populate the rendered chart

<script>
  $.ajax({
    type: "GET",
    url: "<url_path>",
    data: {
        "startDate": startDate,
        "endDate": endDate,
    },
    dataType: 'json',
    statusCode: {
      200: function (data) {
        var chart = Highcharts.charts.find(c => c.renderTo.id == "chart_id");
        // This is line of code where parsing the JS literal is an issue.
        chart.addSeries(JSON.parse(data.seriesData));
      }
    },
  });
</script>

Right now, setting turboThreshold to 0 seems to work for me, or maybe there's a better way to handle dynamically adding and removing series which I may not be aware of? This is not a big issue for me so far so please feel free to close this issue.

[hcpchris]

Thanks, @abbas-mohd56 : That's super helpful! I'll leave the issue open for the moment, but I'll need to look into a pattern that can be used to include a JS literal within the JsonResponse (I vaguely recall there being a pattern to do so in Django, but will need to refresh my memory / do a little searching + testing). I'll leave this issue open for the moment, but will provide some more guidance once I've had a chance to do so some research within the next day or two.