Update flusurv schema and docs with new age, sex, and race groups #1287
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@korlaxxalrok Tentatively tagging you for review. If you're not familiar with these files, let me know and I'll ask around for an alternative! :) |
nmdefries
commented
Sep 15, 2023
melange396
approved these changes
Sep 26, 2023
|
Kudos, SonarCloud Quality Gate passed!
|
|
This got lost somehow... Im going to merge it now; I just need to remember to run the migration before release deployment! |
|
aw crap, i wasnt paying attention and i thought this was going to merge into the |
4 tasks
nmdefries
added a commit
that referenced
this pull request
Oct 29, 2025
* initial switch to new API endpoint; doesn't account for format change * separate fetch fn for whole api obj * get_current_issue to use existing json response * cleanup names and comments - rename input arg to `update` to avoid reassignment later - comment and reuse args_insert - spelling - comment magic constant used in output format - rename location-network/catchmentid map * store extracted rates by strata name rather than position Previously, age strata were numbered sequentially which allowed us to store rate values by position in a list. With the introduction of the new strata, this system is not robust enough to track all the different groups (e.g. ageids are no longer sequential and there are now race and sex groupings with separate numbering systems). * define function to convert json obs to dict grouped by location and epiweek * auto-map from valueids to ordinal and label-based group names * add new strata to sql insert statement by name, not order * pass seasonids around to use in requests for location-specific data * include old and new example API responses * flusurv tests * pass metadata around to reduce API calls * add season label as a descriptive column * move example API responses to make it clear they are not for prod use * add new columns to flusurv endpoint spec * add new age, sex, race strata, and season to schema * list new signals in documentation * move new season column to end of table * add migration script to add new fields to flusurv table * add field descriptions; reformat table so don't need to match line lengths * point from acquisition script to docs and schema def to deduplicate * review cleanup * capitalize constant max_age * move paren to subtract from # of dates Co-authored-by: melange396 <[email protected]> * move n_exected_groups and big groupid comment to global * convert asserts to if-raise * for robust group checking, list all expected groups and do set diff * separate data fetching fns into metadata and location classes * programmatically create location-networkid map; create constants file * switch missing data assert to exception * mock metadata and api objs for tests * stop doing recent season filtering Catchment metadata is not updated with most recent season, so doing this will stop us from fetching data. * warn when no location data returned * test metadata attributes together * correct local module name to load classes from * map some long race groups to abbreviated names * programmatically create sql query to avoid ordering issues * return empty result with right format when no data available * warning prefix * add tests for geo-abbr conversion and location fetching * don't error for missing groups -- some are expected * fn to add empty sex breakdowns to UT * with UT sex subsets getting filled, error on missing groups again * Update flusurv schema and docs with new age, sex, and race groups (#1287) * add new columns to flusurv endpoint spec * add new age, sex, race strata, and season to schema * list new signals in documentation * move new season column to end of table * add migration script to add new fields to flusurv table * add field descriptions; reformat table so don't need to match line lengths * point from acquisition script to docs and schema def to deduplicate * Revert "Update flusurv schema and docs with new age, sex, and race groups" (#1426) This reverts commit 9de7d64. * add new age rate signals * add new columns to flusurv endpoint spec * add schema migration script * move base url to constants * add max age as CLI arg * add flu a/b to DB table * hardcode valueid->signal name; add flu signals * remove ordinal conditional blocks from _groupid_to_name definition * add back female col to sql schema * make error types more specific * add new cols to integration test * remove trailing comma * specify flusurv return fields * warn on unexpected or missing keys and fill in * use unexpected key's ids as suffix; drop UT sex group handling * cast ids to str * drop trailing location * comments and cleanup * cast CLI args to types * note format issue in reference JSONs; rename metadata methods for clarity * filter weeks with user-provided age threshold * add season field to test --------- Co-authored-by: melange396 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Update
flusurvtable schema to support #1278. Also make surerate_age_5,6, and7are returned.Necessary for #1247
Prerequisites:
devbranchdev