Skip to content

metadata generation improvements #990

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
melange396 opened this issue Oct 4, 2022 · 2 comments
Open

metadata generation improvements #990

melange396 opened this issue Oct 4, 2022 · 2 comments
Labels
acquisition changes acquisition logic api change affect the API and its responses enhancement help wanted

Comments

@melange396
Copy link
Collaborator

  • metadata generation routine can get the list of src/sig faster now by using just the signal dimension TABLE and not the big complicated VIEW to gather them
  • add sort_keys=True to json.dumps() to ensure deterministic output
  • add number of points to metadata for each aggregation since we are in there already... and maybe avg number of re-issues to metadata if we wanna also hit the full table

https://github.com/cmu-delphi/delphi-epidata/blob/dev/src/acquisition/covidcast/database.py#L494

@melange396 melange396 added enhancement api change affect the API and its responses acquisition changes acquisition logic labels Oct 6, 2022
@melange396
Copy link
Collaborator Author

this should also include adding a "minimum issue" as described in #232

@melange396
Copy link
Collaborator Author

Another idea:

Use the active attribute of signals from delphi.epidata.server.endpoints.covidcast_utils.model.data_sources to limit metadata generation to only the signals that are currently being updated (similar to how the one metadata endpoint filters "active" signals). The downside is that this presumes the signals spreadsheet is updated and accurate, and is in sync with the local CSV version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
acquisition changes acquisition logic api change affect the API and its responses enhancement help wanted
Projects
None yet
Development

No branches or pull requests

1 participant