Pydantic version conflict between medcat and spacy

[bauerem]

I get a dependency conflict between the pydantic versions required by medcat and spacy:

$ pip3 install zensols.dsprov
...
ERROR: Cannot install medcat and zensols.mednlp because these package versions have conflicting dependencies.

The conflict is caused by:
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.4.1 depends on pydantic!=1.8, !=1.8.1, <1.10.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.4.0 depends on pydantic!=1.8, !=1.8.1, <1.10.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.3.3 depends on pydantic!=1.8, !=1.8.1, <1.9.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.3.2 depends on pydantic!=1.8, !=1.8.1, <1.9.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.3.1 depends on pydantic!=1.8, !=1.8.1, <1.9.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.3.0 depends on pydantic!=1.8, !=1.8.1, <1.9.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.2.6 depends on pydantic!=1.8, !=1.8.1, <1.9.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.2.5 depends on pydantic!=1.8, !=1.8.1, <1.9.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.2.4 depends on pydantic!=1.8, !=1.8.1, <1.9.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.2.3 depends on pydantic!=1.8, !=1.8.1, <1.9.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.2.2 depends on pydantic!=1.8, !=1.8.1, <1.9.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.2.1 depends on pydantic!=1.8, !=1.8.1, <1.9.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.2.0 depends on pydantic!=1.8, !=1.8.1, <1.9.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.1.7 depends on pydantic!=1.8, !=1.8.1, <1.9.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.1.6 depends on pydantic!=1.8, !=1.8.1, <1.9.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.1.5 depends on pydantic!=1.8, !=1.8.1, <1.9.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.1.4 depends on pydantic!=1.8, !=1.8.1, <1.9.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.1.3 depends on pydantic!=1.8, !=1.8.1, <1.9.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.1.2 depends on pydantic!=1.8, !=1.8.1, <1.9.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.1.1 depends on pydantic!=1.8, !=1.8.1, <1.9.0 and >=1.7.4
medcat 1.7.0 depends on pydantic>=1.10.0
spacy 3.1.0 depends on pydantic!=1.8, !=1.8.1, <1.9.0 and >=1.7.4

To fix this you could try to:

1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

[plandes]

Please try:

pip3 install --use-deprecated=legacy-resolver zensols.dsprov

and let me know if it works.

[plandes]

I looked a little more into this. The deprecated legacy resolver is needed for spaCy. I have updated the installation instructions. Note that you should use Python 3.10. (I'll eventually upgrade to 3.11).

Please take a look at that and let me know if that does the trick.

[bauerem]

The command installs most packages. However, there are still some conflicts.

The stacktrace:

ERROR: pip's legacy dependency resolver does not consider dependency conflicts when selecting packages. This behaviour is the source of the following dependency conflicts.
pathy 0.11.0 requires smart-open<7.0.0,>=5.2.1, but you'll have smart-open 7.0.4 which is incompatible.
spacy 3.2.6 requires smart-open<7.0.0,>=5.2.1, but you'll have smart-open 7.0.4 which is incompatible.
spacy 3.2.6 requires typing-extensions<4.6.0,>=3.7.4.1, but you'll have typing-extensions 4.10.0 which is incompatible.
aiohttp 3.8.3 requires charset-normalizer<3.0,>=2.0, but you'll have charset-normalizer 3.3.2 which is incompatible.
datasets 2.18.0 requires fsspec[http]<=2024.2.0,>=2023.1.0, but you'll have fsspec 2024.3.1 which is incompatible.
medcat 1.7.4 requires pydantic>=1.10.0, but you'll have pydantic 1.8.2 which is incompatible.

But also:
Successfully installed MarkupSafe-2.1.5 aiofiles-23.2.1 aiohttp-3.8.3 aiosignal-1.3.1 asttokens-2.4.1 async-timeout-4.0.3 attrs-23.2.0 blis-0.7.11 catalogue-2.0.10 cloudpickle-3.0.0 comm-0.2.2 configparser-5.2.0 ctakes-parser-0.1.0 cycler-0.12.1 cymem-2.0.8 datasets-2.18.0 decorator-5.1.1 dill-0.3.8 exceptiongroup-1.2.0 executing-2.0.1 fonttools-4.50.0 frozendict-2.4.0 frozenlist-1.4.1 future-1.0.0 hyperopt-0.2.7 interlap-0.2.7 ipython-8.22.2 ipywidgets-8.1.2 jedi-0.19.1 jinja2-3.1.3 jsonpickle-3.0.3 jupyterlab-widgets-3.0.10 kiwisolver-1.4.5 langcodes-3.3.0 lxml-4.9.4 matplotlib-3.5.3 matplotlib-inline-0.1.6 medcat-1.7.4 msgpack-1.0.8 msgpack-numpy-0.4.8 multidict-6.0.5 multiprocess-0.70.16 murmurhash-1.0.10 networkx-3.2.1 pandas-1.4.4 parse-1.20.1 parso-0.8.3 pathlib-abc-0.1.1 pathy-0.11.0 patool-1.15.0 pexpect-4.9.0 preshed-3.0.9 prompt-toolkit-3.0.43 protobuf-3.20.3 psutil-5.9.8 psycopg2-2.8.6 ptyprocess-0.7.0 pure-eval-0.2.2 py4j-0.10.9.7 pyarrow-15.0.2 pyarrow-hotfix-0.6 pydantic-1.8.2 pygments-2.17.2 pyparsing-3.1.2 python-dateutil-2.9.0.post0 pytz-2024.1 sentence-transformers-2.2.2 six-1.16.0 spacy-3.2.6 spacy-legacy-3.0.12 spacy-loggers-1.0.5 srsly-2.4.8 stack-data-0.6.3 tabulate-0.8.10 thinc-8.0.17 traitlets-5.14.2 typer-0.4.2 wasabi-0.10.1 wcwidth-0.2.13 widgetsnbextension-4.0.10 xlsxwriter-3.0.9 xxhash-3.4.1 yarl-1.9.4 zensols.datdesc-0.0.2 zensols.db-1.0.0 zensols.dbpg-1.0.0 zensols.deeplearn-1.7.0 zensols.deepnlp-1.9.1 zensols.dsprov-0.0.1 zensols.install-0.2.2 zensols.mednlp-1.3.2 zensols.mimic-1.3.1 zensols.mimicsid-1.3.1 zensols.nlp-1.7.3 zensols.util-1.12.6

(I don't yet have access to the database, so I can't test by running the examples.)

[plandes]

Hi @bauerem,

The top portion of the message is a warning about incompatible package versions found by pip. Are you starting from a new virtual environment or where there already packages installed in this environment? As you can see, pip still installed the packages but these version differences could cause issues.

Regarding the database, you have to register and take a course. Once you download the MIMIC-III database, follow the instructions to create the MIMIC-III SQLite database.

I have tried to automate as much of this process as possible. But there are a lot of moving parts, dependencies, and some bureaucracy (PhysioNet w/MIMIC-III data I'm not allowed to distribute). Try this and then let me know how it goes.