Trying to install on m1

[jrybinski]

Does anyone try to install psycopg2cffi on mac m1? psycopg2 works fine for me. But for psycopg2cffi I got this error

pip3 install psycopg2cffi

      ld: warning: dylib (/opt/homebrew/lib/libpq.dylib) was built for newer macOS version (12.0) than being linked (11.0)
      ld: warning: ignoring file /opt/homebrew/lib/libpq.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
      Traceback (most recent call last):
        File "/private/var/folders/v3/zqxmc7t138n9_qn7n38680dm0000gn/T/pip-install-swb5z_gs/psycopg2cffi_25da3ba0978a417b8f4df8a82525c22f/psycopg2cffi/_impl/libpq.py", line 2, in <module>
          from psycopg2cffi._impl._libpq import ffi, lib as libpq
      ModuleNotFoundError: No module named 'psycopg2cffi._impl._libpq'

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "env3/lib/python3.9/site-packages/cffi/vengine_cpy.py", line 155, in load_library
          module = imp.load_dynamic(self.verifier.get_module_name(),
        File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/imp.py", line 342, in load_dynamic
          return _load(spec)
        File "<frozen importlib._bootstrap>", line 711, in _load
        File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
        File "<frozen importlib._bootstrap>", line 565, in module_from_spec
        File "<frozen importlib._bootstrap_external>", line 1108, in create_module
        File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
      ImportError: dlopen(/private/var/folders/v3/zqxmc7t138n9_qn7n38680dm0000gn/T/pip-install-swb5z_gs/psycopg2cffi_25da3ba0978a417b8f4df8a82525c22f/psycopg2cffi/_impl/__pycache__/_cffi__x857b2ef6x67c52184.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_PQbackendPID'

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/v3/zqxmc7t138n9_qn7n38680dm0000gn/T/pip-install-swb5z_gs/psycopg2cffi_25da3ba0978a417b8f4df8a82525c22f/setup.py", line 126, in <module>
          setup(**setup_kwargs)
        File "/env3/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py", line 108, in setup
          _setup_distribution = dist = klass(attrs)
        File "/env3/lib/python3.9/site-packages/setuptools/dist.py", line 423, in __init__
          _Distribution.__init__(self, {
        File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 292, in __init__
          self.finalize_options()
        File "/env3/lib/python3.9/site-packages/setuptools/dist.py", line 695, in finalize_options
          ep(self)
        File "/env3/lib/python3.9/site-packages/setuptools/dist.py", line 702, in _finalize_setup_keywords
          ep.load()(self, ep.name, value)
        File "/env3/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 219, in cffi_modules
          add_cffi_module(dist, cffi_module)
        File "/env3/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 49, in add_cffi_module
          execfile(build_file_name, mod_vars)
        File "/env3/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 25, in execfile
          exec(code, glob, glob)
        File "psycopg2cffi/_impl/_build_libpq.py", line 171, in <module>
          _config = PostgresConfig()
        File "psycopg2cffi/_impl/_build_libpq.py", line 22, in __init__
          from psycopg2cffi import _config
        File "/private/var/folders/v3/zqxmc7t138n9_qn7n38680dm0000gn/T/pip-install-swb5z_gs/psycopg2cffi_25da3ba0978a417b8f4df8a82525c22f/psycopg2cffi/__init__.py", line 4, in <module>
          from psycopg2cffi import extensions
        File "/private/var/folders/v3/zqxmc7t138n9_qn7n38680dm0000gn/T/pip-install-swb5z_gs/psycopg2cffi_25da3ba0978a417b8f4df8a82525c22f/psycopg2cffi/extensions.py", line 39, in <module>
          from psycopg2cffi._impl import connection as _connection
        File "/private/var/folders/v3/zqxmc7t138n9_qn7n38680dm0000gn/T/pip-install-swb5z_gs/psycopg2cffi_25da3ba0978a417b8f4df8a82525c22f/psycopg2cffi/_impl/connection.py", line 10, in <module>
          from psycopg2cffi._impl import exceptions
        File "/private/var/folders/v3/zqxmc7t138n9_qn7n38680dm0000gn/T/pip-install-swb5z_gs/psycopg2cffi_25da3ba0978a417b8f4df8a82525c22f/psycopg2cffi/_impl/exceptions.py", line 12, in <module>
          from psycopg2cffi._impl.libpq import libpq, ffi
        File "/private/var/folders/v3/zqxmc7t138n9_qn7n38680dm0000gn/T/pip-install-swb5z_gs/psycopg2cffi_25da3ba0978a417b8f4df8a82525c22f/psycopg2cffi/_impl/libpq.py", line 5, in <module>
          libpq = ffi.verify(C_SOURCE, **C_SOURCE_KWARGS)
        File "/env3/lib/python3.9/site-packages/cffi/api.py", line 468, in verify
          lib = self.verifier.load_library()
        File "/env3/lib/python3.9/site-packages/cffi/verifier.py", line 106, in load_library
          return self._load_library()
        File "/env3/lib/python3.9/site-packages/cffi/verifier.py", line 217, in _load_library
          return self._vengine.load_library()
        File "/env3/lib/python3.9/site-packages/cffi/vengine_cpy.py", line 159, in load_library
          raise VerificationError(error)
      cffi.VerificationError: importing '/private/var/folders/v3/zqxmc7t138n9_qn7n38680dm0000gn/T/pip-install-swb5z_gs/psycopg2cffi_25da3ba0978a417b8f4df8a82525c22f/psycopg2cffi/_impl/__pycache__/_cffi__x857b2ef6x67c52184.cpython-39-darwin.so': dlopen(/private/var/folders/v3/zqxmc7t138n9_qn7n38680dm0000gn/T/pip-install-swb5z_gs/psycopg2cffi_25da3ba0978a417b8f4df8a82525c22f/psycopg2cffi/_impl/__pycache__/_cffi__x857b2ef6x67c52184.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_PQbackendPID'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[patrys]

Same problem with the arch mismatch:

ld: warning: ignoring file /opt/homebrew/lib/postgresql@14/libpq.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64

I'm trying to build the package on an M1 Mac running the nightly version of PyPy3.9 that introduced support for M1.

[selevit]

There is no postgres driver for M1 macs, but it's possible to install it under rosetta. Just ensure that during the installation from pip, it is linked with the x86_64 version of the postgres driver. If there is already an m1 driver installed into the system, you would better compile it from source for x86 and put into an isolated prefix and install psycopg2cffi linking with the correct driver version.

An example of a script that can do this (ofk should be run in the x86_64 mode).

#!/bin/bash

if [ $# -le 0 ]; then
  echo "Usage: $0 <PATH_TO_PYTHON_BIN>"
  exit 1
fi
PYTHON=$1

set -ex -o pipefail

PREFIX=${HOME}/.postgresql-14.2-x86_64

curl --fail https://ftp.postgresql.org/pub/source/v14.2/postgresql-14.2.tar.bz2 -o postgresql-14.2.tar.bz2
tar xpf postgresql-14.2.tar.bz2
cd postgresql-14.2
arch -arch x86_64 ./configure --prefix=${PREFIX}
arch -arch x86_64 make
make install

${PYTHON} -mensurepip
CPPFLAGS=-I${PREFIX}/include LDFLAGS=-L${PREFIX}/lib ${PYTHON} -mpip install psycopg2cffi

Tested with pypy3.8 just now.

$ ./compile-psycopg2cffi-from-source.sh

... 

Installing collected packages: six, psycopg2cffi
    Running setup.py install for psycopg2cffi ... done
Successfully installed psycopg2cffi-2.9.0 six-1.16.0

$ pypy

[PyPy 7.3.9 with GCC Apple LLVM 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>> import psycopg2cffi
>>>>

[patrys]

There is no postgres driver for M1 macs, but it's possible to install it under rosetta.

I’m not sure what you mean by that. There is an entire PostrgreSQL stack installed and building psycopg2cffi under cPython works as expected.

The issue here is that everything on M1 is supposed to be arm64 but for whatever reason under PyPy it attempts to build for x86_64 which prevents the linker from working (the warning about mixing architectures and ignoring libpq).

Is your example from an M1 machine?

[patrys]

Ignore my comments in this thread, I was accidentally testing it in the wrong virtualenv that was running an x86_64 build of PyPy. It installs and imports fine when using the correct architecture:

Python 3.9.12 (d4b7ddecdfcdfd39bb680b9ac3de9c658af098bb, Jul 19 2022, 01:30:22)
[PyPy 7.3.10-alpha0 with GCC Apple LLVM 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``pypy is the nuclear fusion of
programming language implementation (pedronis)''
>>>> import platform
>>>> platform.platform()
'macOS-12.5.1-arm64-arm-64bit'
>>>> import psycopg2cffi
>>>>

[selevit]

I just realised that I confused something. At the moment when I used this script, there were no pypy for m1 available (they introduced it only in July 2022).

So, I ran this on m1 on rosetta, yes, and there was an issues with architecture mismatch during linking, because m1 version of postgres was installed into the system.

Now this should just work on pypy on m1, i guess.