Using RDS to read/write/store data is slow and not portable. We may want to switch the data pipeline or dashboard to being written in Python in the future, so using a format that Python can easily read would be preferable.
Looking at a comparison of different formats and packages, storing as a CSV and reading/writing using data.table via fread and fwrite seems pretty good. feather could also be an option (is supported in both R and Python) but it's not clear if standard dataframe and dplyr procedures work with it. data.table data can be seamlessly processed either with dplyr or with faster data.table syntax.
Using RDS to read/write/store data is slow and not portable. We may want to switch the data pipeline or dashboard to being written in Python in the future, so using a format that Python can easily read would be preferable.
Looking at a comparison of different formats and packages, storing as a CSV and reading/writing using
data.tableviafreadandfwriteseems pretty good.feathercould also be an option (is supported in both R and Python) but it's not clear if standard dataframe anddplyrprocedures work with it.data.tabledata can be seamlessly processed either withdplyror with fasterdata.tablesyntax.