Filtering data to build a new dataframe

[derrickcchow]

Hello,

I am creating a graphical dashboard using pandas and Plotly Dash. What I am trying to do is give the end user full control of the data. This allows them to be able to filter down hierarchies and dates to get a snapshot of the data to build a graph using Plotly express. I am Trying to use Vaex to replace Pandas because of the increase performance with large datasets. When filtering the data for time for a bar graph that contains over 180 categories Pandas takes about 18 seconds while Vaex takes 5 minutes. I am unsure that I am using the full potential that Vaex has to offer. I am hoping that using the columns of some of the data("String") as categorical will speed up the filtering of the data. If I Encode the categorical columns and turn the values into integers is there a way to extract the value back to a string?

Derrick

[maartenbreddels]

Yes, that should be the default in version 5, but otherwise call the (undocumented) df._future() to get this behaviour, this should hide that you are working with categorical columns.