Deprecation Warnings for execution on HPC

[cbueth]

Hey, I wanted to reproduce the results of your paper and analyze further cities.
Until now I was not able to analyze data and would be happy to receive support. I am working on a HPC and not with the notebooks.
After populating the parameters/cities.csv with only one city for the start I set up the environment like the README.md instructs. There certificates get superseded back, I guess this is fine.

The following packages will be SUPERSEDED by a higher-priority channel:

  ca-certificates    conda-forge::ca-certificates-2021.10.~ --> anaconda::ca-certificates-2020.10.14-0

When running the scripts/download.job with slurm in code/setup.py the packages gdal and osr are not found. That is why I changed it to be imported from osgeo as

from osgeo import gdal
from osgeo import osr

Is this valid? After this change Shapely (version 1.8.0) throws warnings multiple times. The version is higher than the one you used (1.7.0).

Cities:   0%|          | 0/1 [00:00<?, ?it/s]
<string>:38: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
Networks:   0%|          | 0/10 [00:00<?, ?it/s]
/home/user/.conda/envs/OSMNX/lib/python3.9/site-packages/osmnx/utils_geo.py:281: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
  for polygon in geometry:

The execution continues and then pyproj (version 3.2.1) throws

Cities:   0%|          | 0/1 [00:00<?, ?it/s]
palma: Creating grid
<string>:110: DeprecationWarning: This function is deprecated. See: https://pyproj4.github.io/pyproj/stable/gotchas.html#upgrading-to-pyproj-2-from-pyproj-1

The version is also higher than the one you have seemed to use (2.6.1.post1).

With ignoring the warnings and continuing to executing the analysis there an error occurs. I guess this occurs because ignoring the warnings of deprecation.

Networks:   0%|          | 0/6 [00:00<?, ?it/s]
Fatal error at src/core/vector.c:1181 : Assertion failed: v->stor_begin != v->end
analysis_all.job: line 16:  7108 Aborted                 ~/.conda/envs/OSMNX/bin/python analysisloop.py

Do you see an obvious mistake or correction?

Creating an environment with the requirements_nonbuiltins.txt also takes the higher package versions as before.
So I created an environment which uses the exact (old) versions given in requirements_nonbuiltins.txt. This also did not result in the awaited behavior but also threw an error in the download process.

Loaded parameters.

Loaded PATH.

/home/user/.conda/envs/OSMNX/lib/python3.9/site-packages/pyproj/__init__.py:89: UserWarning: pyproj unable to set database path.
  _pyproj_global_context_initialize()
Setup finished.

Loaded functions.

Running 01.py
Cities:   0%|          | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/user/BikeGrowth/bikenwgrowth/scripts/downloadloop.py", line 11, in <module>
    exec(open("01.py").read())
  File "<string>", line 3, in <module>
  File "/home/user/.conda/envs/OSMNX/lib/python3.9/site-packages/osmnx/geocoder.py", line 101, in geocode_to_gdf
    gdf.crs = settings.default_crs
  File "/home/user/.conda/envs/OSMNX/lib/python3.9/site-packages/geopandas/geodataframe.py", line 199, in __setattr__
    super().__setattr__(attr, val)
  File "/home/user/.conda/envs/OSMNX/lib/python3.9/site-packages/pandas/core/generic.py", line 5500, in __setattr__
    return object.__setattr__(self, name, value)
  File "/home/user/.conda/envs/OSMNX/lib/python3.9/site-packages/geopandas/geodataframe.py", line 423, in crs
    self.geometry.values.crs = value
  File "/home/user/.conda/envs/OSMNX/lib/python3.9/site-packages/geopandas/array.py", line 336, in crs
    self._crs = None if not value else CRS.from_user_input(value)
  File "/home/user/.conda/envs/OSMNX/lib/python3.9/site-packages/pyproj/crs/crs.py", line 479, in from_user_input
    return cls(value, **kwargs)
  File "/home/user/.conda/envs/OSMNX/lib/python3.9/site-packages/pyproj/crs/crs.py", line 326, in __init__
    self._local.crs = _CRS(self.srs)
  File "pyproj/_crs.pyx", line 2352, in pyproj._crs._CRS.__init__
pyproj.exceptions.CRSError: Invalid projection: epsg:4326: (Internal Proj Error: proj_create: no database context specified)

What needs to be changed for the code to run as intended?

[mszell]

Thx for trying to use it!

1. I think you shouldn't use downloadloop.py serverside - OSMNX throws too many connection issues and I never managed to run that download part in a stable way.

2. Concerning the CRSError, it looks like there is no crs context specified. I don't know what exactly the problem is, but this comes up in line 259 of functions.py:

if "crs" not in G.graph:
        G.graph["crs"] = 'epsg:4326' # needed for OSMNX's graph_to_gdfs in utils_graph.py

I hope this can help somehow.

Apart from that, yes, always use exactly the same versions, especially with geo-libraries - they are quite finicky.

[cbueth]

Thank you for your answer!

1. To my delight downloadloop.py worked totally fine on the HPC only with the same DeprecationWarnings as on the HPC. Did you also had them?
   I ran the notebooks 01 and 02 locally (also with same versions as in requirements_nonbuiltins.txt) and transferred the cities data to the HPC but got the same error when trying to analyze the city on the HPC. To compare what would happen locally, I ran notebooks 03 and 04. For 04 the kernel just died at the same point the HPC environment does.
   To check that my city is not the cause I retried with Lyon on the HPC (only) but got the same response.
2. I remember that in 02.py (line 110, 111) transformations are made with projections previously defined, which seem to trigger the warning shapely throws. But the function should still work with shapely 1.7.0. Does this have anything to do with it?

[mszell]

Sorry, I don't really know.