Evaluate alternatives to pandas-datareader for downloading World Bank data

[spjuhel]

Is your feature request related to a problem? Please describe.
The pandas-datareader package has not been updated since 2021-07-13 and is now breaking with Python 3.12 and newer versions due to the removal of distutils. This package is currently used in util.finance.py for downloading World Bank data and in related tests.

Describe the solution you'd like
Identify and migrate to an alternative for downloading World Bank data. The steps could include:

1. Researching viable replacements for pandas-datareader, such as wbdata (see here) or direct API usage via libraries like requests or pandas.
2. Refactoring util.finance.py to use the new solution.
3. Updating related tests to ensure compatibility with the new implementation.

Describe alternatives you've considered

• Continuing to use pandas-datareader by forking and maintaining a private version. However, this would increase the maintenance burden on the project.
• Temporarily patching the current implementation to support Python 3.12 and beyond, though this is not a sustainable long-term solution.

For reference, a related issue in pandas-datareader highlights these problems: pydata/pandas-datareader#977

[peanutfun]

Check if the data is JSON. pandas.read_json supports URLs, other read functions might as well

[spjuhel]

Note: Update the litpop tutorial here:

Regarding the GDP (nominal GDP at current USD) and income group values, they are obtained from the World Bank using the pandas-datareader API. If a value is missing, the value of the closest year is considered. When no values are provided from the World Bank, we use the Natural Earth repository values.

[peanutfun]

Researching viable replacements for pandas-datareader, such as wbdata (see here) or direct API usage via libraries like requests or pandas.

wbdata actually looks quite nice. It has a small set of dependencies and even provides a persistent cache for the downloaded data.

The World Bank data is queried via an API, so building our own solution via requests can be a bit cumbersome. I tried directly accessing the data through pandas.read_json using an API request URL, but that is not straightforward. So I think we should go with wbdata.

[spjuhel]

I see you are assigned, are you implementing this then? Happy to review it once done if that is the case :)

[peanutfun]

Found a solution without using wbdata, see #1033