Clean up delphi_utils (take 2) #2014

melange396 · 2024-08-05T14:49:37Z

Description

migrating setup.py to pyproject.toml and cleaning up dependencies

Changelog

Itemize code/test/documentation changes and files added/removed.

added pyproject.toml

Associated Issue(s)

Addresses Clean up delphi_utils #1973

(this is a redo of #1984, which was reverted in #2001 due to problems with Jenkins builds)

Co-authored-by: Dmitry Shemetov <[email protected]>

dshemetov

I didn't catch in the previous PR that the individual indicators don't actually need the dev version of the delphi utils, so made those suggestions. If the indicators have dependency issues because of these changes, we should add the dependencies to the indicator's dependency list (seems like good form to isolate dev dependencies).

changehc/Makefile

claims_hosp/Makefile

doctor_visits/Makefile

google_symptoms/Makefile

hhs_hosp/Makefile

nssp/Makefile

nwss_wastewater/Makefile

quidel_covidtest/Makefile

quidel_covidtest/setup.py

sir_complainsalot/Makefile

_delphi_utils_python/pyproject.toml

aysim319 · 2024-08-07T14:18:48Z

I didn't catch in the previous PR that the individual indicators don't actually need the dev version of the delphi utils, so made those suggestions. If the indicators have dependency issues because of these changes, we should add the dependencies to the indicator's dependency list (seems like good form to isolate dev dependencies).

most of these need for mock, the only other package that has mock as a requirement in setup.py is google symptoms. But google symptom linter acted weird when I took the dev from the makefile off. see comment

aysim319 · 2024-08-07T15:44:32Z

Do we also want to convert the rest of the indicator to pyproject.toml? maybe putting in the template_python to starting thing off?

dshemetov · 2024-08-07T21:27:47Z

most of these need for mock

Right, I'm saying we should add mock to each indicator's individual setup.py instead of grabbing it from _delphi_utils' dev dependencies. Sounds good on migrating the indicators to pyproject.toml, then each indicator can have its own "dev" set of packages and we can put mock in there.

Also, please add requests to the _delphi_utils pyproject.toml, it was in my suggestion above.

aysim319 · 2024-08-07T21:42:01Z

Oh I didn't see it before 🤦‍♀️Thanks for pointing it out

dshemetov · 2024-08-07T21:58:05Z

FYI, this Jenkins script will need to be adjusted to check for the presence of pyproject.toml instead.

aysim319 · 2024-08-08T15:30:13Z

most of these need for mock

Right, I'm saying we should add mock to each indicator's individual setup.py instead of grabbing it from _delphi_utils' dev dependencies. Sounds good on migrating the indicators to pyproject.toml, then each indicator can have its own "dev" set of packages and we can put mock in there.

Okay just making sure I'm understanding this properly; do we want to convert the existing indicators also to pyproject.toml with the same setup as delphi_utils that has an optional requirement in this PR or do we want to punt that for another PR? if we are punting to another PR, I feel like it makes more sense to leave it as is. Especially since this is
a step 2 of the bigger cleaning up delphi_utils (#1973)

dshemetov · 2024-08-08T17:40:38Z

I don't have a strong preference. Getting delphi_utils cleaned is the key task here and this PR is close to taking care of that (I still think we should remove the ".[dev]" from the individual indicators and debug this CI). Porting the indicators to pyproject.toml is nice to have for consistency, but not critical. If I had to pick, I'd say just move the indicators in another PR.

* pandas[excel]<2 doesn't work, go back to xlrd and comment * add mock to changehc * add requests * remove setup.py from template

dshemetov

This is good to go. Let's make an issue to migrate the indicators to pyproject.toml and link to this thread. Let's remember to squash merge, so the commit log gets cleaned.

* doc: add logger examples to README and format * deployment + staging details * wrap lines * Update _template_python/INDICATOR_DEV_GUIDE.md Co-authored-by: David Weber <[email protected]> * Update _template_python/INDICATOR_DEV_GUIDE.md Co-authored-by: David Weber <[email protected]> * Update _template_python/INDICATOR_DEV_GUIDE.md Co-authored-by: David Weber <[email protected]> * Update INDICATOR_DEV_GUIDE.md with mysql syntax * Update INDICATOR_DEV_GUIDE.md reorder instruction on cron jobs * switch from input_file to input_dir (#1995) * adjust nssp max_age * add pyproject.toml * made changes based on suggestion * update ci config * separate new line Co-authored-by: Dmitry Shemetov <[email protected]> * fix: add explicit boto and moto dependencies to indicators * trying to get package data to show * installing dev package before testing in ci * adding dependency * move comment to right package * Update _delphi_utils_python/pyproject.toml Co-authored-by: Dmitry Shemetov <[email protected]> * Revert "Clean up delphi_utils " * refactoring date parsing to seperate file * first implimentation * small cleanup and adding missing data * changed wording and some logic * made test more explicit * add missing file * cleaning up tests * consolidating tests into one * fixed wording * consolidating test/fixture * clean up test * Update google_symptoms/delphi_google_symptoms/patch.py remove old code; not used anymore Co-authored-by: nmdefries <[email protected]> * clean up lint * remove teardown * cleaning and renaming * more explicit branching for regular run vs patching, renaming, cleaning * cleanup and rewrite logic * avoid linking keywords in PR template * Update _delphi_utils_python/README.md Co-authored-by: george <[email protected]> * Refactor Jenkins pipeline - Remove usage of parallel in the build and package stages - Retain usage of parallel in the deploy stages - Separate build and package into separate scripts to support build-only stages - Add a build-only stage for dev/feature branches * Add separate build and package scripts * Update script comments * Remove unused script * changes based on suggestion * standardizing output dir in test * changing back to session scope * more logging and update params * adding back conditional for num_export_days * test_run does not like fixtures * make deep copies of params fixtures * linting * test patch check if num_export_days is set or not * patch fn takes params as an arg, like run_module * list values added using actual day as an example * move pull date creation into pull_gs_data * don't modify num_export_days; define start/end date in terms of it and lag * refer to params by name; patch_flag -> custom_run flag * use test data based on recent actual indicator run * say where gold test data came from * not all date_utils tests need metadata * formatting * fn docs * lint and cleanup * adding more tests * removed extra lag and made test more robust * export_start_date is optional since the first available date is baked in * need to account for historical / current run for adding lag * adding validation script * fix typo * fixing typo in test * lint * sample run for validation * remove unused input Co-authored-by: nmdefries <[email protected]> * remove unused import * added progress chunk to limit logging (#2024) * added progress chunk to limit logging * lint * comment based on suggestion * add hhs geo level (#2036) * Add discussion of patching functionality to indicator manual (#2031) * basic patching info * custom_run * add hhs to geo section * de-emphasize R in contributing doc, and add links * ref/issue date intro * naming and force push standards * secret and params links * Clean up delphi_utils (take 2) (#2014) * add pyproject.toml * made changes based on suggestion * update ci config * separate new line Co-authored-by: Dmitry Shemetov <[email protected]> * fix: add explicit boto and moto dependencies to indicators * trying to get package data to show * installing dev package before testing in ci * adding dependency * move comment to right package * Update _delphi_utils_python/pyproject.toml Co-authored-by: Dmitry Shemetov <[email protected]> * more suggestion * adding pyproject.toml to template * Update changehc/Makefile * Update claims_hosp/Makefile * Update doctor_visits/Makefile * Update google_symptoms/Makefile * Update hhs_hosp/Makefile * Update nchs_mortality/Makefile * Update nssp/Makefile * Update nwss_wastewater/Makefile * Update quidel_covidtest/Makefile * Update sir_complainsalot/Makefile * fix: dependence bugs and cleanup * pandas[excel]<2 doesn't work, go back to xlrd and comment * add mock to changehc * add requests * remove setup.py from template * build: fix Jenkins build script --------- Co-authored-by: Amaris Sim <[email protected]> Co-authored-by: aysim319 <[email protected]> Co-authored-by: Dmitry Shemetov <[email protected]> * fix: build-indicators.sh looks for setup.py and pyproject.toml (#2048) * fix: build-indicators.sh looks for setup.py again Reverting a premature change. * fix: readability and future-proof * fix: more readability * chore: bump delphi_utils to 0.3.25 * chore: bump covidcast-indicators to 0.3.56 * [create-pull-request] automated change --------- Co-authored-by: Dmitry Shemetov <[email protected]> Co-authored-by: minhkhul <[email protected]> Co-authored-by: Nat DeFries <[email protected]> Co-authored-by: minhkhul <[email protected]> Co-authored-by: David Weber <[email protected]> Co-authored-by: george <[email protected]> Co-authored-by: Amaris Sim <[email protected]> Co-authored-by: aysim319 <[email protected]> Co-authored-by: Brian Clark <[email protected]> Co-authored-by: Delphi Deploy Bot <[email protected]> Co-authored-by: minhkhul <[email protected]>

aysim319 and others added 10 commits August 5, 2024 10:45

add pyproject.toml

69571ef

made changes based on suggestion

7936340

update ci config

341f3b1

separate new line

4a58492

Co-authored-by: Dmitry Shemetov <[email protected]>

fix: add explicit boto and moto dependencies to indicators

cbc7453

trying to get package data to show

d88dba1

installing dev package before testing in ci

46496ce

adding dependency

4129c98

move comment to right package

c920b55

Update _delphi_utils_python/pyproject.toml

068d4d0

Co-authored-by: Dmitry Shemetov <[email protected]>

dshemetov requested changes Aug 5, 2024

View reviewed changes

dshemetov mentioned this pull request Aug 6, 2024

Clean up delphi_utils #1973

Open

3 tasks

dshemetov reviewed Aug 6, 2024

View reviewed changes

_delphi_utils_python/pyproject.toml Outdated Show resolved Hide resolved

aysim319 force-pushed the 1973-clean-up-delphi-utils-take-2 branch from d99598e to b7f07c0 Compare August 7, 2024 14:12

aysim319 force-pushed the 1973-clean-up-delphi-utils-take-2 branch from 5ba012d to 068d4d0 Compare August 7, 2024 14:21

aysim319 added 2 commits August 7, 2024 11:36

more suggestion

b80b3ea

adding pyproject.toml to template

9fd531e

dshemetov added 6 commits August 8, 2024 10:42

Update changehc/Makefile

755b671

Update claims_hosp/Makefile

8f35aad

Update doctor_visits/Makefile

bff5bb8

Update google_symptoms/Makefile

1e1ccaf

Update hhs_hosp/Makefile

e95b5d7

Update nchs_mortality/Makefile

4bc32d8

dshemetov added 4 commits August 8, 2024 10:42

Update nssp/Makefile

1d8b1d4

Update nwss_wastewater/Makefile

cd44d90

Update quidel_covidtest/Makefile

c2fdae5

Update sir_complainsalot/Makefile

45cfd8f

dshemetov force-pushed the 1973-clean-up-delphi-utils-take-2 branch 4 times, most recently from 4c63558 to ada86c2 Compare August 8, 2024 19:00

fix: dependence bugs and cleanup

46ef13b

* pandas[excel]<2 doesn't work, go back to xlrd and comment * add mock to changehc * add requests * remove setup.py from template

dshemetov force-pushed the 1973-clean-up-delphi-utils-take-2 branch from ada86c2 to 46ef13b Compare August 8, 2024 19:03

dshemetov approved these changes Aug 8, 2024

View reviewed changes

aysim319 mentioned this pull request Aug 9, 2024

migrate setup.py to pyproject.toml #2016

Open

dshemetov mentioned this pull request Aug 9, 2024

build+ci: make local, gh, and Jenkins envs identical #2019

Open

aysim319 mentioned this pull request Aug 12, 2024

migrate setup.py to pyproject.toml #2021

Closed

build: fix Jenkins build script

e952850

aysim319 merged commit 86399ad into main Sep 9, 2024
16 checks passed

This was referenced Sep 11, 2024

Release covidcast-indicators 0.3.56 [ABANDONED] #2047

Closed

fix: build-indicators.sh looks for setup.py and pyproject.toml #2048

Merged

Release covidcast-indicators 0.3.56 [ROLLED BACK] #2049

Merged

melange396 mentioned this pull request Sep 12, 2024

Release covidcast-indicators 0.3.56 #2052

Merged

nmdefries deleted the 1973-clean-up-delphi-utils-take-2 branch December 10, 2024 22:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clean up delphi_utils (take 2) #2014

Clean up delphi_utils (take 2) #2014

melange396 commented Aug 5, 2024

dshemetov left a comment •

edited

Loading

aysim319 commented Aug 7, 2024 •

edited

Loading

aysim319 commented Aug 7, 2024

dshemetov commented Aug 7, 2024 •

edited

Loading

aysim319 commented Aug 7, 2024 •

edited

Loading

dshemetov commented Aug 7, 2024 •

edited

Loading

aysim319 commented Aug 8, 2024 •

edited

Loading

dshemetov commented Aug 8, 2024 •

edited

Loading

dshemetov left a comment •

edited

Loading

Clean up delphi_utils (take 2) #2014

Clean up delphi_utils (take 2) #2014

Conversation

melange396 commented Aug 5, 2024

Description

Changelog

Associated Issue(s)

dshemetov left a comment • edited Loading

Choose a reason for hiding this comment

aysim319 commented Aug 7, 2024 • edited Loading

aysim319 commented Aug 7, 2024

dshemetov commented Aug 7, 2024 • edited Loading

aysim319 commented Aug 7, 2024 • edited Loading

dshemetov commented Aug 7, 2024 • edited Loading

aysim319 commented Aug 8, 2024 • edited Loading

dshemetov commented Aug 8, 2024 • edited Loading

dshemetov left a comment • edited Loading

Choose a reason for hiding this comment

dshemetov left a comment •

edited

Loading

aysim319 commented Aug 7, 2024 •

edited

Loading

dshemetov commented Aug 7, 2024 •

edited

Loading

aysim319 commented Aug 7, 2024 •

edited

Loading

dshemetov commented Aug 7, 2024 •

edited

Loading

aysim319 commented Aug 8, 2024 •

edited

Loading

dshemetov commented Aug 8, 2024 •

edited

Loading

dshemetov left a comment •

edited

Loading