Clean up delphi_utils

[melange396]

delphi_utils is a big hodge-podge of a package with a lot of dependencies, thus it has a relatively large net footprint and takes a while to install. Lets see if we can lessen that.

• Move packages in setup.py that are unnecessary for a downstream user but still used in test/lint/ci, like: darker, freezegun, mock, moto, pydocstyle, pylint, pytest-cov, pytest, requests-mock. They can be put into something like a "requirements.test.txt" file that gets installed via the CI and/or Makefile.
• Remove from setup.py any unused packages, as gitpython appears to be.
• The covidcast package has its own wacky imports, which includes a bunch of imaging/graphics and geospatial libraries. It is only used in the delphi_utils.validator module for pulling data from the API and can be replaced with the lighter-weight delphi_epidata client instead. Doing that is not quite as easy as moving requirements entries around, and it is covered in in replace the python covidcast client in validator #1972.

[melange396]

We could even put some of the esoteric usage and/or larger installation dependencies behind package "extras".

[dshemetov]

If we combine this with migrating over to pyproject.toml, we could use the optional-dependencies interface to separate out the dev dependencies. See here for an example of migrating over to pyproject.toml. Then, we could change Makefile in _delphi_utils_python to do pip install -e .[dev] and all the rest of the indicators Makefiles can stay the same.

[melange396]

migrating to using pyproject.toml helps unlock this issue: #1839

[dshemetov]

Gonna reopen this issue. Also, gonna edit your OP @melange396 so the subtasks are clearer because #1984 addressed the first of the subtasks and we could probably tackle the second in the Jenkins-fix follow-up too. The third subtask can be closed when we finish up #1972.

[dshemetov]

Thinking about the unused package subtask, I found deptry and ran it on _delphi_utils and it reported this:

(env) ➜  _delphi_utils_python git:(1972-replace-covidcast) ✗ deptry .
Scanning 31 files...

data_proc/geomap/geo_data_proc.py:13:8: DEP003 'requests' imported but it is a transitive dependency
data_proc/geomap/source-file-sanity-check.ipynb:23:8: DEP003 'matplotlib' imported but it is a transitive dependency
delphi_utils/flash_eval/eval_day.py:6:1: DEP003 'scipy' imported but it is a transitive dependency
delphi_utils/validator/datafetcher.py:13:8: DEP003 'requests' imported but it is a transitive dependency
requirements.txt: DEP002 'scs' defined as a dependency but not used in the codebase
requirements.txt: DEP002 'darker' defined as a dependency but not used in the codebase
requirements.txt: DEP002 'freezegun' defined as a dependency but not used in the codebase
requirements.txt: DEP002 'mock' defined as a dependency but not used in the codebase
requirements.txt: DEP002 'moto' defined as a dependency but not used in the codebase
requirements.txt: DEP002 'pydocstyle' defined as a dependency but not used in the codebase
requirements.txt: DEP002 'pylint' defined as a dependency but not used in the codebase
requirements.txt: DEP002 'pytest-cov' defined as a dependency but not used in the codebase
requirements.txt: DEP002 'pytest' defined as a dependency but not used in the codebase
requirements.txt: DEP002 'requests-mock' defined as a dependency but not used in the codebase
requirements.txt: DEP002 'xlrd' defined as a dependency but not used in the codebase
Found 16 dependency issues.

For more information, see the documentation: https://deptry.com/

The following will be handled by #2014, as they get moved to dev.

requirements.txt: DEP002 'darker' defined as a dependency but not used in the codebase
requirements.txt: DEP002 'freezegun' defined as a dependency but not used in the codebase
requirements.txt: DEP002 'mock' defined as a dependency but not used in the codebase
requirements.txt: DEP002 'moto' defined as a dependency but not used in the codebase
requirements.txt: DEP002 'pydocstyle' defined as a dependency but not used in the codebase
requirements.txt: DEP002 'pylint' defined as a dependency but not used in the codebase
requirements.txt: DEP002 'pytest-cov' defined as a dependency but not used in the codebase
requirements.txt: DEP002 'pytest' defined as a dependency but not used in the codebase
requirements.txt: DEP002 'requests-mock' defined as a dependency but not used in the codebase

• xlrd is needed by Pandas (maybe we should write pandas[excel] so this dependence is explicit? EDIT: pandas==1.5.3 does not support that extra).
• scs is explicitly version pinned for cvxpy because of upstream issues
• seems like a good idea to explicitly list requests as a dependecy (done in follow up)
• flash we can probably ignore until needed and geomap scripts we can ignore until needed