316 use etnservice as a dependency and drop sql helpers #317
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PietrH
force-pushed
the
316-use-etnservice-as-a-dependency-and-drop-_sql-helpers
branch
from
0e15606 to
ad753e0
Compare
…s' of github.com:inbo/etn into 316-use-etnservice-as-a-dependency-and-drop-_sql-helpers
|
Should we have a seperate test file that checks if the API and SQL return the same results? Or should these tests be included in the function test files? I'm quite tempted to use: for(function_to_test in getNamespaceExports("etnservice")) {
test_that(paste(
function_to_test,
"returns same result over api as over local db connection"
),
{
skip_if_offline()
api_result <- do.call(function_to_test, list(api = TRUE))
sql_result <- do.call(function_to_test, list(api = FALSE))
expect_identical(api_result, sql_result, label = function_to_test)
})
}
But not all functions run without arguments (returned objects too large?) |
|
Maybe this is a good use case for a custom expectation. For example, you could create a It could then be called as part of a regular test-function.R file: testthat("get_animals() returns same result locally and via API", {
expect_call_agnostic(get_animals(animal_id = "5"))
}) |
|
That sounds great! Do you think we should include that here, or rather as a separate PR? |
|
9 functions currently fail to return the same result over the api as over a local connection. |
I'll leave that to you |
This was referenced Oct 18, 2024
…s' of github.com:inbo/etn into 316-use-etnservice-as-a-dependency-and-drop-_sql-helpers
…s' of github.com:inbo/etn into 316-use-etnservice-as-a-dependency-and-drop-_sql-helpers
…s' of github.com:inbo/etn into 316-use-etnservice-as-a-dependency-and-drop-_sql-helpers
|
Lots of test failures on the RStudio Server: ==> devtools::test()
Loading etn
Testing etn
✔ | F W S OK | Context
✔ | 16 | api_helpers [2.1s]
✔ | 10 | check_value
✔ | 2 2 | connect_to_etn
⠇ | 1 2 1 5 | download_acoustic_dataset Downloading data to directory `/tmp/Rtmp0BNJm8`:
* (1/6): downloading animals.csv
* (2/6): downloading tags.csv
* (3/6): downloading detections.csv
* (4/6): downloading deployments.csv
* (5/6): downloading receivers.csv
* (6/6): adding datapackage.json as file metadata
Summary statistics for dataset `2014_demer`:
* number of animals: 16
* number of tags: 16
* number of detections: 236918
* number of deployments: 1152
* number of receivers: 249
* first date of detection: 2014-04-18
* last date of detection: 2018-09-15
* included scientific names: Petromyzon marinus, Rutilus rutilus, Silurus glanis, Squalius cephalus
* included acoustic projects: V2LCHASES, albert, demer, dijle, zeeschelde
✖ | 4 2 1 21 | download_acoustic_dataset [92.5s]
───────────────────────────────────────────────────────────────────────────────────────
Warning (test-download_acoustic_dataset.R:27:3): download_acoustic_dataset() creates the expected files
Adding new snapshot for variant 'api':
Code
download_acoustic_dataset(api = TRUE, animal_project_code = "2014_demer",
directory = download_dir)
Message
Downloading data to directory :
* (1/6): downloading animals.csv
* (2/6): downloading tags.csv
* (3/6): downloading detections.csv
* (4/6): downloading deployments.csv
* (5/6): downloading receivers.csv
* (6/6): adding datapackage.json as file metadata
Summary statistics for dataset `2014_demer`:
* number of animals: 16
* number of tags: 16
* number of detections: 236918
* number of deployments: 1152
* number of receivers: 249
* first date of detection: 2014-04-18
* last date of detection: 2018-09-15
* included scientific names: Petromyzon marinus, Rutilus rutilus, Silurus glanis, Squalius cephalus
* included acoustic projects: V2LCHASES, albert, demer, dijle, zeeschelde
Error (test-download_acoustic_dataset.R:52:3): download_acoustic_dataset() creates the expected messages and files using local db
Error in `files_to_create %in% list.files(tempdir())`: object 'files_to_create' not found
Backtrace:
▆
1. ├─testthat::expect_true(all(files_to_create %in% list.files(tempdir()))) at test-download_acoustic_dataset.R:52:3
2. │ └─testthat::quasi_label(enquo(object), label, arg = "object")
3. │ └─rlang::eval_bare(expr, quo_get_env(quo))
4. └─files_to_create %in% list.files(tempdir())
Warning (test-download_acoustic_dataset.R:63:3): download_acoustic_dataset() returns message and summary stats
Adding new snapshot:
Code
cat(download_acoustic_dataset(animal_project_code = "2014_demer"))
Message
Downloading data to directory `2014_demer`:
* (1/6): downloading animals.csv
* (2/6): downloading tags.csv
* (3/6): downloading detections.csv
* (4/6): downloading deployments.csv
* (5/6): downloading receivers.csv
* (6/6): adding datapackage.json as file metadata
Summary statistics for dataset `2014_demer`:
* number of animals: 16
* number of tags: 16
* number of detections: 236918
* number of deployments: 1152
* number of receivers: 249
* first date of detection: 2014-04-18
* last date of detection: 2018-09-15
* included scientific names: Petromyzon marinus, Rutilus rutilus, Silurus glanis, Squalius cephalus
* included acoustic projects: V2LCHASES, albert, demer, dijle, zeeschelde
Failure (test-download_acoustic_dataset.R:119:3): download_acoustic_dataset() creates a valid Frictionless Data Package
Expected `suppressMessages(frictionless::read_resource(datapackage, "detections"))` to run without any warnings.
i Actually got a <vroom_parse_issue> with text:
One or more parsing issues, call `problems()` on your data frame for details, e.g.:
dat <- vroom(...)
problems(dat)
Failure (test-download_acoustic_dataset.R:144:3): download_acoustic_dataset() returns CSV files with expected columns
fetch_schema_fields(datapackage, "detections") has length 21, not length 20.
Failure (test-download_acoustic_dataset.R:192:3): download_acoustic_dataset() returns CSV files with columns in expected order
sapply(fetch_schema_fields(datapackage, "detections"), function(x) x[["name"]]) (`actual`) not identical to names(...) (`expected`).
actual | expected
[10] "station_name" | "station_name" [10]
[11] "deploy_latitude" | "deploy_latitude" [11]
[12] "deploy_longitude" | "deploy_longitude" [12]
[13] "depth_in_meters" -
[14] "sensor_value" | "sensor_value" [13]
[15] "sensor_unit" | "sensor_unit" [14]
[16] "sensor2_value" | "sensor2_value" [15]
───────────────────────────────────────────────────────────────────────────────────────
✔ | 70 | get_acoustic_deployments [44.7s]
✖ | 1 4 70 | get_acoustic_detections [106.0s]
───────────────────────────────────────────────────────────────────────────────────────
Failure (test-get_acoustic_detections.R:59:3): get_acoustic_detections() returns the expected columns
names(df) (`actual`) not identical to `expected_col_names` (`expected`).
actual | expected
[10] "station_name" | "station_name" [10]
[11] "deploy_latitude" | "deploy_latitude" [11]
[12] "deploy_longitude" | "deploy_longitude" [12]
- "depth_in_meters" [13]
[13] "sensor_value" | "sensor_value" [14]
[14] "sensor_unit" | "sensor_unit" [15]
[15] "sensor2_value" | "sensor2_value" [16]
───────────────────────────────────────────────────────────────────────────────────────
✔ | 14 | get_acoustic_projects [14.6s]
✔ | 18 | get_acoustic_receivers [21.5s]
✔ | 14 | get_animal_projects [20.0s]
✔ | 39 | get_animals [57.7s]
✔ | 14 | get_cpod_projects [17.6s]
✔ | 2 | get_credentials
✔ | 36 | get_tags [47.6s]
✔ | 6 | list_acoustic_project_codes
✔ | 6 | list_acoustic_tag_ids
✖ | 3 3 | list_animal_ids
───────────────────────────────────────────────────────────────────────────────────────
Failure (test-list_animal_ids.R:14:3): list_animal_ids() returns same list over api and sql
`result_api` (`actual`) not identical to `result_sql` (`expected`).
`actual[10566:10573]`: 24245 24246 24247 55431 55433 57292 57293 57294
`expected[10566:10571]`: 24245 24246 24247 57292 57293 57294
actual | expected
[18463] 74212 | 74212 [18461]
[18464] 74213 | 74213 [18462]
[18465] 74214 | 74214 [18463]
[18466] 74220 - 74609 [18464]
[18467] 74221 - 74616 [18465]
[18468] 74222 - 76528 [18466]
[18469] 74223 - 76534 [18467]
[18470] 74224 - 76967 [18468]
[18471] 74225 - 76973 [18469]
[18472] 74226 - 76978 [18470]
... ... ... and 4646 more ...
Failure (test-list_animal_ids.R:24:3): list_animal_ids returns at least 5 known values
all(known_ids %in% result_api) is not TRUE
`actual`: FALSE
`expected`: TRUE
Failure (test-list_animal_ids.R:25:3): list_animal_ids returns at least 5 known values
all(known_ids %in% result_sql) is not TRUE
`actual`: FALSE
`expected`: TRUE
───────────────────────────────────────────────────────────────────────────────────────
✔ | 6 | list_animal_project_codes
✔ | 12 | list_cpod_project_codes
✔ | 8 | list_deployment_ids
✔ | 4 | list_receiver_ids
✔ | 4 | list_scientific_names
✔ | 4 | list_station_names
✔ | 8 | list_tag_serial_numbers
⠴ | 16 | list_values 4 unique chr_col values
3 unique num_col values
✔ | 19 | list_values
✔ | 1 | utils [1.4s]
✖ | 3 9 | write_dwc [45.8s]
───────────────────────────────────────────────────────────────────────────────────────
Error (test-write_dwc.R:7:3): [SQL] write_dwc() can write csv files to a path
<Rcpp::not_compatible/C++Error/error/condition>
Error: Expecting a single string value: [type=character; extent=0].
Backtrace:
▆
1. ├─base::suppressMessages(...) at test-write_dwc.R:7:3
2. │ └─base::withCallingHandlers(expr, message = function(c) invokeRestart("muffleMessage"))
3. └─etn::write_dwc(...)
4. └─etn:::conduct_parent_to_helpers(api, json = FALSE, ignored_arguments = "directory") at etn/R/write_dwc.R:59:3
5. ├─base::do.call(...) at etn/R/conduct_parent_to_helpers.R:47:5
6. └─etnservice (local) `<fn>`(...)
7. ├─DBI::dbGetQuery(connection, dwc_occurrence_sql) at etnservice/R/write_dwc.R:98:3
8. └─DBI::dbGetQuery(connection, dwc_occurrence_sql)
9. └─odbc (local) .local(conn, statement, ...)
10. ├─DBI::dbSendQuery(...)
11. └─odbc::dbSendQuery(...)
12. └─odbc (local) .local(conn, statement, ...)
13. └─odbc:::OdbcResult(...)
14. └─odbc:::new_result(p = connection@ptr, sql = statement, immediate = immediate)
Error (test-write_dwc.R:20:3): [SQL] write_dwc() can return data as list of tibbles rather than files
<Rcpp::not_compatible/C++Error/error/condition>
Error: Expecting a single string value: [type=character; extent=0].
Backtrace:
▆
1. ├─base::suppressMessages(...) at test-write_dwc.R:20:3
2. │ └─base::withCallingHandlers(expr, message = function(c) invokeRestart("muffleMessage"))
3. └─etn::write_dwc(...)
4. └─etn:::conduct_parent_to_helpers(api, json = FALSE, ignored_arguments = "directory") at etn/R/write_dwc.R:59:3
5. ├─base::do.call(...) at etn/R/conduct_parent_to_helpers.R:47:5
6. └─etnservice (local) `<fn>`(...)
7. ├─DBI::dbGetQuery(connection, dwc_occurrence_sql) at etnservice/R/write_dwc.R:98:3
8. └─DBI::dbGetQuery(connection, dwc_occurrence_sql)
9. └─odbc (local) .local(conn, statement, ...)
10. ├─DBI::dbSendQuery(...)
11. └─odbc::dbSendQuery(...)
12. └─odbc (local) .local(conn, statement, ...)
13. └─odbc:::OdbcResult(...)
14. └─odbc:::new_result(p = connection@ptr, sql = statement, immediate = immediate)
Error (test-write_dwc.R:31:3): [SQL] write_dwc() returns the expected Darwin Core terms as columns
<Rcpp::not_compatible/C++Error/error/condition>
Error: Expecting a single string value: [type=character; extent=0].
Backtrace:
▆
1. ├─base::suppressMessages(...) at test-write_dwc.R:31:3
2. │ └─base::withCallingHandlers(expr, message = function(c) invokeRestart("muffleMessage"))
3. └─etn::write_dwc(...)
4. └─etn:::conduct_parent_to_helpers(api, json = FALSE, ignored_arguments = "directory") at etn/R/write_dwc.R:59:3
5. ├─base::do.call(...) at etn/R/conduct_parent_to_helpers.R:47:5
6. └─etnservice (local) `<fn>`(...)
7. ├─DBI::dbGetQuery(connection, dwc_occurrence_sql) at etnservice/R/write_dwc.R:98:3
8. └─DBI::dbGetQuery(connection, dwc_occurrence_sql)
9. └─odbc (local) .local(conn, statement, ...)
10. ├─DBI::dbSendQuery(...)
11. └─odbc::dbSendQuery(...)
12. └─odbc (local) .local(conn, statement, ...)
13. └─odbc:::OdbcResult(...)
14. └─odbc:::new_result(p = connection@ptr, sql = statement, immediate = immediate)
───────────────────────────────────────────────────────────────────────────────────────
Maximum number of failures exceeded; quitting at end of file.
ℹ Increase this number with (e.g.) `testthat::set_max_fails(Inf)`
══ Results ════════════════════════════════════════════════════════════════════════════
Duration: 473.3 s
── Skipped tests (7) ──────────────────────────────────────────────────────────────────
• BUG #278 crashes on big queries like demer and dijle (1):
test-get_acoustic_detections.R:319:3
• BUG #319: tests will always fail after first deprecation test (2):
test-connect_to_etn.R:8:3, test-connect_to_etn.R:15:3
• Issue #283 detection_id is currently not unique (1):
test-get_acoustic_detections.R:27:3
• TODO: https://github.com/inbo/etn/issues/216 (1):
test-get_acoustic_detections.R:443:3
• TODO: re-enable after https://github.com/inbo/etn/issues/252 (1):
test-get_acoustic_detections.R:423:3
• empty test (1): test-download_acoustic_dataset.R:11:1
── Failed tests ───────────────────────────────────────────────────────────────────────
Error (test-download_acoustic_dataset.R:52:3): download_acoustic_dataset() creates the expected messages and files using local db
Error in `files_to_create %in% list.files(tempdir())`: object 'files_to_create' not found
Backtrace:
▆
1. ├─testthat::expect_true(all(files_to_create %in% list.files(tempdir()))) at test-download_acoustic_dataset.R:52:3
2. │ └─testthat::quasi_label(enquo(object), label, arg = "object")
3. │ └─rlang::eval_bare(expr, quo_get_env(quo))
4. └─files_to_create %in% list.files(tempdir())
Failure (test-download_acoustic_dataset.R:119:3): download_acoustic_dataset() creates a valid Frictionless Data Package
Expected `suppressMessages(frictionless::read_resource(datapackage, "detections"))` to run without any warnings.
i Actually got a <vroom_parse_issue> with text:
One or more parsing issues, call `problems()` on your data frame for details, e.g.:
dat <- vroom(...)
problems(dat)
Failure (test-download_acoustic_dataset.R:144:3): download_acoustic_dataset() returns CSV files with expected columns
fetch_schema_fields(datapackage, "detections") has length 21, not length 20.
Failure (test-download_acoustic_dataset.R:192:3): download_acoustic_dataset() returns CSV files with columns in expected order
sapply(fetch_schema_fields(datapackage, "detections"), function(x) x[["name"]]) (`actual`) not identical to names(...) (`expected`).
actual | expected
[10] "station_name" | "station_name" [10]
[11] "deploy_latitude" | "deploy_latitude" [11]
[12] "deploy_longitude" | "deploy_longitude" [12]
[13] "depth_in_meters" -
[14] "sensor_value" | "sensor_value" [13]
[15] "sensor_unit" | "sensor_unit" [14]
[16] "sensor2_value" | "sensor2_value" [15]
Failure (test-get_acoustic_detections.R:59:3): get_acoustic_detections() returns the expected columns
names(df) (`actual`) not identical to `expected_col_names` (`expected`).
actual | expected
[10] "station_name" | "station_name" [10]
[11] "deploy_latitude" | "deploy_latitude" [11]
[12] "deploy_longitude" | "deploy_longitude" [12]
- "depth_in_meters" [13]
[13] "sensor_value" | "sensor_value" [14]
[14] "sensor_unit" | "sensor_unit" [15]
[15] "sensor2_value" | "sensor2_value" [16]
Failure (test-list_animal_ids.R:14:3): list_animal_ids() returns same list over api and sql
`result_api` (`actual`) not identical to `result_sql` (`expected`).
`actual[10566:10573]`: 24245 24246 24247 55431 55433 57292 57293 57294
`expected[10566:10571]`: 24245 24246 24247 57292 57293 57294
actual | expected
[18463] 74212 | 74212 [18461]
[18464] 74213 | 74213 [18462]
[18465] 74214 | 74214 [18463]
[18466] 74220 - 74609 [18464]
[18467] 74221 - 74616 [18465]
[18468] 74222 - 76528 [18466]
[18469] 74223 - 76534 [18467]
[18470] 74224 - 76967 [18468]
[18471] 74225 - 76973 [18469]
[18472] 74226 - 76978 [18470]
... ... ... and 4646 more ...
Failure (test-list_animal_ids.R:24:3): list_animal_ids returns at least 5 known values
all(known_ids %in% result_api) is not TRUE
`actual`: FALSE
`expected`: TRUE
Failure (test-list_animal_ids.R:25:3): list_animal_ids returns at least 5 known values
all(known_ids %in% result_sql) is not TRUE
`actual`: FALSE
`expected`: TRUE
Error (test-write_dwc.R:7:3): [SQL] write_dwc() can write csv files to a path
<Rcpp::not_compatible/C++Error/error/condition>
Error: Expecting a single string value: [type=character; extent=0].
Backtrace:
▆
1. ├─base::suppressMessages(...) at test-write_dwc.R:7:3
2. │ └─base::withCallingHandlers(expr, message = function(c) invokeRestart("muffleMessage"))
3. └─etn::write_dwc(...)
4. └─etn:::conduct_parent_to_helpers(api, json = FALSE, ignored_arguments = "directory") at etn/R/write_dwc.R:59:3
5. ├─base::do.call(...) at etn/R/conduct_parent_to_helpers.R:47:5
6. └─etnservice (local) `<fn>`(...)
7. ├─DBI::dbGetQuery(connection, dwc_occurrence_sql) at etnservice/R/write_dwc.R:98:3
8. └─DBI::dbGetQuery(connection, dwc_occurrence_sql)
9. └─odbc (local) .local(conn, statement, ...)
10. ├─DBI::dbSendQuery(...)
11. └─odbc::dbSendQuery(...)
12. └─odbc (local) .local(conn, statement, ...)
13. └─odbc:::OdbcResult(...)
14. └─odbc:::new_result(p = connection@ptr, sql = statement, immediate = immediate)
Error (test-write_dwc.R:20:3): [SQL] write_dwc() can return data as list of tibbles rather than files
<Rcpp::not_compatible/C++Error/error/condition>
Error: Expecting a single string value: [type=character; extent=0].
Backtrace:
▆
1. ├─base::suppressMessages(...) at test-write_dwc.R:20:3
2. │ └─base::withCallingHandlers(expr, message = function(c) invokeRestart("muffleMessage"))
3. └─etn::write_dwc(...)
4. └─etn:::conduct_parent_to_helpers(api, json = FALSE, ignored_arguments = "directory") at etn/R/write_dwc.R:59:3
5. ├─base::do.call(...) at etn/R/conduct_parent_to_helpers.R:47:5
6. └─etnservice (local) `<fn>`(...)
7. ├─DBI::dbGetQuery(connection, dwc_occurrence_sql) at etnservice/R/write_dwc.R:98:3
8. └─DBI::dbGetQuery(connection, dwc_occurrence_sql)
9. └─odbc (local) .local(conn, statement, ...)
10. ├─DBI::dbSendQuery(...)
11. └─odbc::dbSendQuery(...)
12. └─odbc (local) .local(conn, statement, ...)
13. └─odbc:::OdbcResult(...)
14. └─odbc:::new_result(p = connection@ptr, sql = statement, immediate = immediate)
Error (test-write_dwc.R:31:3): [SQL] write_dwc() returns the expected Darwin Core terms as columns
<Rcpp::not_compatible/C++Error/error/condition>
Error: Expecting a single string value: [type=character; extent=0].
Backtrace:
▆
1. ├─base::suppressMessages(...) at test-write_dwc.R:31:3
2. │ └─base::withCallingHandlers(expr, message = function(c) invokeRestart("muffleMessage"))
3. └─etn::write_dwc(...)
4. └─etn:::conduct_parent_to_helpers(api, json = FALSE, ignored_arguments = "directory") at etn/R/write_dwc.R:59:3
5. ├─base::do.call(...) at etn/R/conduct_parent_to_helpers.R:47:5
6. └─etnservice (local) `<fn>`(...)
7. ├─DBI::dbGetQuery(connection, dwc_occurrence_sql) at etnservice/R/write_dwc.R:98:3
8. └─DBI::dbGetQuery(connection, dwc_occurrence_sql)
9. └─odbc (local) .local(conn, statement, ...)
10. ├─DBI::dbSendQuery(...)
11. └─odbc::dbSendQuery(...)
12. └─odbc (local) .local(conn, statement, ...)
13. └─odbc:::OdbcResult(...)
14. └─odbc:::new_result(p = connection@ptr, sql = statement, immediate = immediate)
[ FAIL 11 | WARN 2 | SKIP 7 | PASS 416 ]
══ Terminated early ═══════════════════════════════════════════════════════════════════
`` |
…s' of github.com:inbo/etn into 316-use-etnservice-as-a-dependency-and-drop-_sql-helpers
|
Deployed a patch to etnservice for depth_in_meters but it isn't present in the deployment (that was successful, I checked). To be continued! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When I first started working on this, I didn't implement the changes to the helper immediately, but rather experimented in a single function: b03e88f, later on I moved this change to the
forward_to_api()helper, meaning that a lot of lines can be removed. All this code still exists of course, just inetnserviceinstead of here as a duplicate.From this PR onward, the API and local database connection should always return exactly the same result, any differences resulting from unintended type parsing should be considered a bug.
It seems not all the changes from v2.3 are present in this branch, namely the vignettes.
remove unused dependencies -> indirect, now moved tomoved to issue v2.3.1: remove dependencies that have now become indirect viaetnserviceetnservice#331API - SQL mismatch
mismatches for:
get_cpod_projects
list_animal_ids: missing animal ID
56314 known id is missing from both the API and SQL result, I've sent an email to Claudia
Merge with v2.2.0
I made a mistake, I should have updated v2.3.1 from main instead.
depth_in_meterscolumn is missing fromget_acoustic_detections()Review comments
utils.Rto their own files, or toutils-api.RUpdate v2.3.1 from main