Skip to content

Commit 024a5a3

Browse files
authored
Merge pull request #1443 from cmu-delphi/release/delphi-epidata-4.1.21
Release Delphi Epidata 4.1.21
2 parents 71b7577 + d05f7fb commit 024a5a3

File tree

19 files changed

+221
-67
lines changed

19 files changed

+221
-67
lines changed

.bumpversion.cfg

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
[bumpversion]
2-
current_version = 4.1.20
2+
current_version = 4.1.21
33
commit = False
44
tag = False
55

@@ -9,10 +9,10 @@ tag = False
99

1010
[bumpversion:file:src/client/delphi_epidata.R]
1111

12+
[bumpversion:file:src/client/delphi_epidata.py]
13+
1214
[bumpversion:file:src/client/packaging/npm/package.json]
1315

1416
[bumpversion:file:src/client/packaging/pypi/setup.py]
1517

16-
[bumpversion:file:src/client/packaging/pypi/delphi_epidata/__init__.py]
17-
1818
[bumpversion:file:dev/local/setup.cfg]

dev/local/setup.cfg

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[metadata]
22
name = Delphi Development
3-
version = 4.1.20
3+
version = 4.1.21
44

55
[options]
66
packages =
@@ -24,6 +24,7 @@ packages =
2424
delphi.epidata.acquisition.twtr
2525
delphi.epidata.acquisition.wiki
2626
delphi.epidata.client
27+
delphi.epidata.common
2728
delphi.epidata.server
2829
delphi.epidata.server.admin
2930
delphi.epidata.server.admin.templates

docs/Gemfile.lock

+5-3
Original file line numberDiff line numberDiff line change
@@ -205,14 +205,14 @@ GEM
205205
rb-fsevent (~> 0.10, >= 0.10.3)
206206
rb-inotify (~> 0.9, >= 0.9.10)
207207
mercenary (0.3.6)
208-
mini_portile2 (2.8.5)
208+
mini_portile2 (2.8.6)
209209
minima (2.5.1)
210210
jekyll (>= 3.5, < 5.0)
211211
jekyll-feed (~> 0.9)
212212
jekyll-seo-tag (~> 2.1)
213213
minitest (5.17.0)
214214
multipart-post (2.1.1)
215-
nokogiri (1.16.2)
215+
nokogiri (1.16.5)
216216
mini_portile2 (~> 2.8.2)
217217
racc (~> 1.4)
218218
octokit (4.20.0)
@@ -225,7 +225,8 @@ GEM
225225
rb-fsevent (0.10.4)
226226
rb-inotify (0.10.1)
227227
ffi (~> 1.0)
228-
rexml (3.2.5)
228+
rexml (3.2.8)
229+
strscan (>= 3.0.9)
229230
rouge (3.26.0)
230231
ruby-enum (0.9.0)
231232
i18n
@@ -242,6 +243,7 @@ GEM
242243
faraday (> 0.8, < 2.0)
243244
simpleidn (0.2.1)
244245
unf (~> 0.1.4)
246+
strscan (3.1.0)
245247
terminal-table (1.8.0)
246248
unicode-display_width (~> 1.1, >= 1.1.1)
247249
thread_safe (0.3.6)

docs/api/covidcast-signals/jhu-csse.md

+5-5
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
title: JHU Cases and Deaths
3-
parent: Data Sources and Signals
3+
parent: Inactive Signals
44
grand_parent: COVIDcast Main Endpoint
55
---
66

@@ -23,12 +23,12 @@ University.
2323

2424
| Signal and 7-day average signal | Description |
2525
|---|---|
26-
| `confirmed_cumulative_num` | Cumulative number of confirmed COVID-19 cases <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
27-
| `confirmed_cumulative_prop` | Cumulative number of confirmed COVID-19 cases per 100,000 population <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
26+
| `confirmed_cumulative_num` and `confirmed_7dav_cumulative_num` | Cumulative number of confirmed COVID-19 cases <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
27+
| `confirmed_cumulative_prop` and `confirmed_7dav_cumulative_prop` | Cumulative number of confirmed COVID-19 cases per 100,000 population <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
2828
| `confirmed_incidence_num` and `confirmed_7dav_incidence_num` | Number of new confirmed COVID-19 cases, daily <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
2929
| `confirmed_incidence_prop` and `confirmed_7dav_incidence_prop` | Number of new confirmed COVID-19 cases per 100,000 population, daily <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
30-
| `deaths_cumulative_num` | Cumulative number of confirmed deaths due to COVID-19 <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
31-
| `deaths_cumulative_prop` | Cumulative number of confirmed due to COVID-19, per 100,000 population <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
30+
| `deaths_cumulative_num` and `deaths_7dav_cumulative_num` | Cumulative number of confirmed deaths due to COVID-19 <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
31+
| `deaths_cumulative_prop` and `deaths_7dav_cumulative_prop` | Cumulative number of confirmed due to COVID-19, per 100,000 population <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
3232
| `deaths_incidence_num` and `deaths_7dav_incidence_num` | Number of new confirmed deaths due to COVID-19, daily <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
3333
| `deaths_incidence_prop` and `deaths_7dav_incidence_prop` | Number of new confirmed deaths due to COVID-19 per 100,000 population, daily <br/> **Earliest date available:** 2020-01-22 & 2020-02-20 |
3434

docs/api/covidcast-signals/nchs-mortality.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ York State in our reports.
6868

6969
We report the NCHS Mortality data in a weekly format (`time_type=week` &
7070
`time_value={YYYYWW}`, where `YYYYWW` refers to an epiweek). The CDC defines
71-
the [epiweek](https://wwwn.cdc.gov/nndss/document/MMWR_Week_overview.pdf) as
71+
the [epiweek](https://web.archive.org/web/20210623224758/https://wwwn.cdc.gov/nndss/document/MMWR_Week_overview.pdf) as
7272
seven days, from Sunday to Saturday. We check the week-ending dates provided in
7373
the NCHS morality data and use Python package
7474
[epiweeks](https://pypi.org/project/epiweeks/) to convert them into epiweek

docs/symptom-survey/publications.md

+18-1
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,23 @@ Pandemic"](https://www.pnas.org/topic/548) in *PNAS*:
2626

2727
Research publications using the survey data include:
2828

29+
- Z. Yang, R. Krishnan, and B. Li (2024). [The interplay between individual
30+
mobility, health risk, and economic choice: A holistic model for COVID-19
31+
policy intervention](https://doi.org/10.1287/ijds.2023.0013). *INFORMS
32+
Journal on Data Science*.
33+
- A. Srivastava, J. M. Ramirez, S. Díaz-Aranda, J. Aguilar, A. F. Anta, A. Ortega,
34+
and R. E. Lillo (2024). [Nowcasting temporal trends using indirect surveys](https://doi.org/10.1609/aaai.v38i20.30242).
35+
In *Proceedings of the 38th AAAI Conference on Artificial Intelligence* 38,
36+
22359–22367.
37+
- P. Porebski, S. Venkatramanan, A. Adiga, B. Klahn, B. Hurt, M. L. Wilson,
38+
J. Chen, A. Vullikanti, M. Marathe & B. Lewis (2024). [Data-driven
39+
mechanistic framework with stratified immunity and effective transmissibility
40+
for COVID-19 scenario projections](https://doi.org/10.1016/j.epidem.2024.100761).
41+
*Epidemics* 100761.
42+
- V. Nelson, B. Bashyal, P.-N. Tan & Y. A. Argyris (2024). [Vaccine rhetoric
43+
on social media and COVID-19 vaccine uptake rates: A triangulation using
44+
self-reported vaccine acceptance](https://doi.org/10.1016/j.socscimed.2024.116775).
45+
*Social Science & Medicine* 116775.
2946
- R.R. Andridge (2024). [Using proxy pattern-mixture models to explain bias in
3047
estimates of COVID-19 vaccine uptake from two large surveys](https://doi.org/10.1093/jrsssa/qnae005).
3148
*Journal of the Royal Statistical Society Series A: Statistics in Society*.
@@ -34,7 +51,7 @@ Research publications using the survey data include:
3451
*IISE Transactions*.
3552
- de Vries, M., Kim, J.Y. & Han, H. (2023). [The unequal landscape of civic
3653
opportunity in America](https://doi.org/10.1038/s41562-023-01743-1). *Nature
37-
Human Behavior*.
54+
Human Behavior* 8, 256-263.
3855
- E. Tuzhilina, T. J. Hastie, D. J. McDonald, J. K. Tay & R. Tibshirani (2023).
3956
[Smooth multi-period forecasting with application to prediction of COVID-19
4057
cases](https://doi.org/10.1080/10618600.2023.2285337). *Journal of Computational

integrations/client/test_delphi_epidata.py

+79
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
# standard library
44
import time
55
from json import JSONDecodeError
6+
from requests.models import Response
67
from unittest.mock import MagicMock, patch
78

89
# first party
@@ -41,6 +42,8 @@ def localSetUp(self):
4142
# use the local instance of the Epidata API
4243
Epidata.BASE_URL = 'http://delphi_web_epidata/epidata'
4344
Epidata.auth = ('epidata', 'key')
45+
Epidata.debug = False
46+
Epidata.sandbox = False
4447

4548
# use the local instance of the epidata database
4649
secrets.db.host = 'delphi_database_epidata'
@@ -221,6 +224,82 @@ def test_retry_request(self, get):
221224
{'result': 0, 'message': 'error: Expecting value: line 1 column 1 (char 0)'}
222225
)
223226

227+
@patch('requests.post')
228+
@patch('requests.get')
229+
def test_debug(self, get, post):
230+
"""Test that in debug mode request params are correctly logged."""
231+
class MockResponse:
232+
def __init__(self, content, status_code):
233+
self.content = content
234+
self.status_code = status_code
235+
def raise_for_status(self): pass
236+
237+
Epidata.debug = True
238+
239+
try:
240+
with self.subTest(name='test multiple GET'):
241+
with self.assertLogs('delphi_epidata_client', level='INFO') as logs:
242+
get.reset_mock()
243+
get.return_value = MockResponse(b'{"key": "value"}', 200)
244+
Epidata._request_with_retry("test_endpoint1", params={"key1": "value1"})
245+
Epidata._request_with_retry("test_endpoint2", params={"key2": "value2"})
246+
247+
output = logs.output
248+
self.assertEqual(len(output), 4) # [request, response, request, response]
249+
self.assertIn("Sending GET request", output[0])
250+
self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/test_endpoint1/\"", output[0])
251+
self.assertIn("\"params\": {\"key1\": \"value1\"}", output[0])
252+
self.assertIn("Received response", output[1])
253+
self.assertIn("\"status_code\": 200", output[1])
254+
self.assertIn("\"len\": 16", output[1])
255+
self.assertIn("Sending GET request", output[2])
256+
self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/test_endpoint2/\"", output[2])
257+
self.assertIn("\"params\": {\"key2\": \"value2\"}", output[2])
258+
self.assertIn("Received response", output[3])
259+
self.assertIn("\"status_code\": 200", output[3])
260+
self.assertIn("\"len\": 16", output[3])
261+
262+
with self.subTest(name='test GET and POST'):
263+
with self.assertLogs('delphi_epidata_client', level='INFO') as logs:
264+
get.reset_mock()
265+
get.return_value = MockResponse(b'{"key": "value"}', 414)
266+
post.reset_mock()
267+
post.return_value = MockResponse(b'{"key": "value"}', 200)
268+
Epidata._request_with_retry("test_endpoint3", params={"key3": "value3"})
269+
270+
output = logs.output
271+
self.assertEqual(len(output), 3) # [request, response, request, response]
272+
self.assertIn("Sending GET request", output[0])
273+
self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/test_endpoint3/\"", output[0])
274+
self.assertIn("\"params\": {\"key3\": \"value3\"}", output[0])
275+
self.assertIn("Received 414 response, retrying as POST request", output[1])
276+
self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/test_endpoint3/\"", output[1])
277+
self.assertIn("\"params\": {\"key3\": \"value3\"}", output[1])
278+
self.assertIn("Received response", output[2])
279+
self.assertIn("\"status_code\": 200", output[2])
280+
self.assertIn("\"len\": 16", output[2])
281+
finally: # make sure this global is always reset
282+
Epidata.debug = False
283+
284+
@patch('requests.post')
285+
@patch('requests.get')
286+
def test_sandbox(self, get, post):
287+
"""Test that in debug + sandbox mode request params are correctly logged, but no queries are sent."""
288+
Epidata.debug = True
289+
Epidata.sandbox = True
290+
try:
291+
with self.assertLogs('delphi_epidata_client', level='INFO') as logs:
292+
Epidata.covidcast('src', 'sig', 'day', 'county', 20200414, '01234')
293+
output = logs.output
294+
self.assertEqual(len(output), 1)
295+
self.assertIn("Sending GET request", output[0])
296+
self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/covidcast/\"", output[0])
297+
get.assert_not_called()
298+
post.assert_not_called()
299+
finally: # make sure these globals are always reset
300+
Epidata.debug = False
301+
Epidata.sandbox = False
302+
224303
def test_geo_value(self):
225304
"""test different variants of geo types: single, *, multi."""
226305

requirements.api.txt

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ delphi_utils==0.3.15
22
epiweeks==2.1.2
33
Flask==2.2.5
44
Flask-Limiter==3.3.0
5-
jinja2==3.1.3
5+
jinja2==3.1.4
66
more_itertools==8.4.0
77
mysqlclient==2.1.1
88
orjson==3.9.15

requirements.dev.txt

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
aiohttp==3.9.2
1+
aiohttp==3.9.4
22
black>=20.8b1
33
bump2version==1.0.1
44
covidcast==0.1.5

src/client/delphi_epidata.R

+1-1
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ Epidata <- (function() {
1515
# API base url
1616
BASE_URL <- getOption('epidata.url', default = 'https://api.delphi.cmu.edu/epidata/')
1717

18-
client_version <- '4.1.20'
18+
client_version <- '4.1.21'
1919

2020
auth <- getOption("epidata.auth", default = NA)
2121

src/client/delphi_epidata.js

+1-1
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222
}
2323
})(this, function (exports, fetchImpl, jQuery) {
2424
const BASE_URL = "https://api.delphi.cmu.edu/epidata/";
25-
const client_version = "4.1.20";
25+
const client_version = "4.1.21";
2626

2727
// Helper function to cast values and/or ranges to strings
2828
function _listitem(value) {

src/client/delphi_epidata.py

+26-10
Original file line numberDiff line numberDiff line change
@@ -10,21 +10,17 @@
1010

1111
# External modules
1212
import requests
13+
import time
1314
import asyncio
1415
from tenacity import retry, stop_after_attempt
1516

1617
from aiohttp import ClientSession, TCPConnector, BasicAuth
17-
from importlib.metadata import version, PackageNotFoundError
1818

19-
# Obtain package version for the user-agent. Uses the installed version by
20-
# preference, even if you've installed it and then use this script independently
21-
# by accident.
22-
try:
23-
_version = version("delphi-epidata")
24-
except PackageNotFoundError:
25-
_version = "0.script"
19+
from delphi.epidata.common.logger import get_structured_logger
2620

27-
_HEADERS = {"user-agent": "delphi_epidata/" + _version + " (Python)"}
21+
__version__ = "4.1.21"
22+
23+
_HEADERS = {"user-agent": "delphi_epidata/" + __version__ + " (Python)"}
2824

2925

3026
class EpidataException(Exception):
@@ -47,7 +43,11 @@ class Epidata:
4743
BASE_URL = "https://api.delphi.cmu.edu/epidata"
4844
auth = None
4945

50-
client_version = _version
46+
client_version = __version__
47+
48+
logger = get_structured_logger('delphi_epidata_client')
49+
debug = False # if True, prints extra logging statements
50+
sandbox = False # if True, will not execute any queries
5151

5252
# Helper function to cast values and/or ranges to strings
5353
@staticmethod
@@ -71,9 +71,25 @@ def _list(values):
7171
def _request_with_retry(endpoint, params={}):
7272
"""Make request with a retry if an exception is thrown."""
7373
request_url = f"{Epidata.BASE_URL}/{endpoint}/"
74+
if Epidata.debug:
75+
Epidata.logger.info("Sending GET request", url=request_url, params=params, headers=_HEADERS, auth=Epidata.auth)
76+
if Epidata.sandbox:
77+
resp = requests.Response()
78+
resp._content = b'true'
79+
return resp
80+
start_time = time.time()
7481
req = requests.get(request_url, params, auth=Epidata.auth, headers=_HEADERS)
7582
if req.status_code == 414:
83+
if Epidata.debug:
84+
Epidata.logger.info("Received 414 response, retrying as POST request", url=request_url, params=params, headers=_HEADERS)
7685
req = requests.post(request_url, params, auth=Epidata.auth, headers=_HEADERS)
86+
if Epidata.debug:
87+
Epidata.logger.info(
88+
"Received response",
89+
status_code=req.status_code,
90+
len=len(req.content),
91+
time=round(time.time() - start_time, 4)
92+
)
7793
# handle 401 and 429
7894
req.raise_for_status()
7995
return req

src/client/packaging/npm/package.json

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
"name": "delphi_epidata",
33
"description": "Delphi Epidata API Client",
44
"authors": "Delphi Group",
5-
"version": "4.1.20",
5+
"version": "4.1.21",
66
"license": "MIT",
77
"homepage": "https://github.com/cmu-delphi/delphi-epidata",
88
"bugs": {

src/client/packaging/pypi/CHANGELOG.md

+32
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,38 @@
33
All notable future changes to the `delphi_epidata` python client will be documented in this file.
44
The format is based on [Keep a Changelog](http://keepachangelog.com/).
55

6+
## [4.1.21] - 2024-05-20
7+
8+
### Includes
9+
- https://github.com/cmu-delphi/delphi-epidata/pull/1418
10+
- https://github.com/cmu-delphi/delphi-epidata/pull/1436
11+
12+
### Added
13+
- Adds two debug flags:
14+
- `debug` logs info about HTTP requests and responses
15+
- `sandbox` prevents any HTTP requests from actually executing, allowing for tests that do not incur server load.
16+
- Fixes the `user-agent` version so that it is correctly set to match the current client release.
17+
18+
## [4.1.17] - 2024-01-30
19+
20+
### Includes
21+
- https://github.com/cmu-delphi/delphi-epidata/pull/1363
22+
23+
### Changed
24+
- Replaced use of deprecated setuptools' `pkg_resources` library with the native `importlib.metadata` library.
25+
26+
## [4.1.13] - 2023-11-04
27+
28+
### Includes
29+
- https://github.com/cmu-delphi/delphi-epidata/pull/1323
30+
- https://github.com/cmu-delphi/delphi-epidata/pull/1330
31+
32+
### Changed
33+
- Appends a trailing slash to URLs requested by the Python client, which should prevent an automatic redirect and an extra request to the server.
34+
35+
### Removed
36+
- Removed the `covidcast_nowcast()` method, as the associated API endpoint is no longer available.
37+
638
## [4.1.11] - 2023-10-12
739

840
### Includes
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
from .delphi_epidata import Epidata
1+
from .delphi_epidata import Epidata, __version__
22

33
name = "delphi_epidata"
4-
__version__ = "4.1.20"

0 commit comments

Comments
 (0)