Testing for network grid

[michaelkirk]

For testing proj's network grid, I want to be able to say "do this transformation without the network grid" and "do this transformation with the network grid, downloading if necessary."

However, currently it's tough to get consistent results locally when testing the network grid.

I suspect this is because the network grid will be downloaded locally the first time only, and then future calls will use the already downloaded grid. Which makes perfect sense for a normal user's behavior, but is bad for tests.

If I recall correctly, the network grid files get downloaded to the user writable directory. I recall as a work-around in the past, I was able to delete this directory every time I ran the test, but that's cumbersome.

Maybe instead there's some way we can use a one-off tmp_dir or something for each test to get this kind of behavior isolated per-test.

[michaelkirk]

Applications may also customize the user writable directory using proj_context_set_user_writable_directory() (since 9.5)

This seems promising!

[urschrei]

Yes it's customisable, and I'm confident that we can supply our own grid file to the test: I'm (obviously) not against integration testing per se, but the requirement for our standard test suite to call out to the network has long bothered me. This way, we can disable network functionality entirely in tests while still testing grid corrections and / or have a separate integration test that performs the network call(s).

[urschrei]

Another possibility that may be causing confusion: libproj ships with a set of grids on homebrew, and proj_context_set_user_writable_directory() appends to the "standard" directories, so depending on what conversions you're testing, you may be getting grid-enabled conversions no matter what you do (nb our CI does not ship them).

Does the test_network_enabled_conversion test pass for you on macOS (assuming you're on macOS)?

[urschrei]

The CI installs don't ship with any grids, so it's easy there…

[michaelkirk]

Hrm. It kind of seems like homebrew just downloads "all the data"

https://github.com/Homebrew/homebrew-core/blob/97a784d0000c867701900760fd0c1999ba135937/Formula/p/proj.rb#L32C10-L32C44

    url "https://github.com/OSGeo/PROJ-data/releases/download/1.24.0/proj-data-1.24.zip"

So maybe testing this on proj installed from brew is not possible?

[urschrei]

That's possible, urgh. What about testing it with a source build?

[michaelkirk]

Using --features bundled_proj network and the tests pass. Thank you for the suggestion!

[michaelkirk]

At some point long ago, I convinced myself that, once downloaded, even offline transformations would use any available already-downloaded grids.

I assume that's why our test has:

    // Disable caching to ensure we're accessing the network.
    // Cache is stored in proj's [user writeable directory](https://proj.org/resource_files.html#user-writable-directory)
    online_builder.grid_cache_enable(false);

Otherwise, we'd persist the grid to ~/Library/Application\ Support/proj/cache.db, which might cause
subsequent runs of the same test to behave differently.

I've verified that no cache.db is written with grid_cache_enabled(false). And, conversely, that cache is persisted when grid_cache_enable(true). So everything is working as expected.

However, even when removing that line, and persisting the cache, the test continues to pass with repeated runs. So it seems like enable_network(false) will prevent the proj instance from looking at network grids - even those previously downloaded into the user_writable_dir. This surprised me!

Based on the behavior of homebrew's proj, which pre-downloads and packages grids to <pkg-dir>/share/proj (not the user_writable_dir) , these "packaged" grids will be read regardless of the online_builder.enable_network setting, so apparently there is a difference between how proj handles these "pre-packaged" grids and the "user downloaded" grids.

I don't think we need to do anything about this, I'm just sharing my understanding since it's a bit touchy.