Local cache

[mcannon068nw]

Similar to how civicpy enables a local cache, we should consider giving dgipy the option to download and pull from a local cache of DGIdb data. Unsure of what exactly this would look like in DGIpy given the difference in data load and structure between this and civic though. From a brief discussion with @acoffman, DGIpy would have a much bigger data set to pull than civicpy but conceivably would pull it much less frequently. Could we do some kind of cache as we go implementation (#30)?

@acoffman @susannasiebert @jsstevenson what would be the best way to go about implementing this?

[jsstevenson]

Pitch for a naive, simple solution: publish a docker image that stands up a DGIdb server, runs the rake task to download + import a DB dump, and spins up graphql service. There's your local cache.

[mcannon068nw]

are we concerned about this potentially introducing another hurdle?

I guess I'm just concerned that someone might have trouble setting up a docker compared to someone just running something like dgipy.prepare_cache() or something.

[mcannon068nw]

alternatively, maybe both should be done. It might be good to have a docker image option in addition to a cache option?

[mcannon068nw]

upon further review docker image seems to be the most straightforward option both for the code base as currently written, and for use cases that would need this feature

[jsstevenson]

my slightly less succinct pitch:

• it'd be nice to do a pure-python local cache a la civicpy
• I think it'd entail some reengineering. Our code currently just queries DGIdb directly and restructures the result. My understanding of the civicpy approach is that it downloads all data, restructures it locally into pure-python data structures, and then offers searches over those structures.
  • We also want to provide an R client with feature parity, so would we need to do this in both Python and R?
• A docker based solution does entail extra complexity on the user side (users gotta run some docker commands). But I would wager that the people working in circumstances where a local cache is most important (critical annotation operations that need reliability and security) would be able to handle that extra complexity.

So, insofar as this is a priority, I think we should push forward with a Docker-based workflow and supporting documentation first -- make sure our data release/download methods are where we want them, write up an image and publish it, write up supporting docs for how to get it running.