Finding external dependencies in non-standard places

[awvwgk]

This is a whole can of worms. The documentation regarding the dependency() function in meson gives a nice impression on all the things one can / has to consider when dealing with dependencies.

How does fpm find dependencies now?
Currently, we assume the compiler can find external libraries and include files all by itself.

This works fine for system libraries, but usually there are packages under different prefixes like /opt or in the users home which could also be used, but are not included by default.

Using a dependency tool
A somewhat standard way to handle non-system dependencies are pkg-config or cmake package files, some platforms might provide their own like OSX frameworks. Also some packages are known to provide incorrect package files.

Using environment variables
An third-party library usually provides a way to make itself known to the system, using environment variables like CPATH to add include directories, LIBRARY_PATH to add to the library search path and LD_LIBRARY_PATH to add to the runtime load path. This mechanism allows fpm to automatically pick up third-party dependencies by relying on the compiler.

Setup scripts / environment modules
Setting those variables in the first place is tricky, usually third-party libraries provide scripts which can be sourced by the shell or an environment module that can be loaded. Sometimes those scripts and modules are not working correctly as well. This is a common issue with environment modules which miss certain crucial environment variables (missing CMAKE_PREFIX_PATH is a classic).

---

As a developer there is nothing more painful than working in a broken environment. Either because some overzealous setup script in /etc/profile.d messes with all other packages, an environment module exports the wrong paths, or a pc file with a typo won't be recognized.

Can we / Should we do something about this?

[LKedward]

Thanks for opening Sebastian @awvwgk and for the detailed summary of issues;
this is indeed a complicated one.

So it seems the environment variables / environment modules approach should already work with fpm since they rely on the compiler?

It seems that a possible next step would be to add automatic support for pkg-config and then cmake like meson, perhaps with a syntax like:

[dependencies]
zlib = { external = "zlib" }

which, like meson, attempts to find the package automatically using the supported methods.

---

Regarding erroneous module files or pc files, I don't think there is very much that we can do in fpm to alleviate this is there?

[ivan-pi]

It seems that a possible next step would be to add automatic support for pkg-config and then cmake like meson

Should this be pursued directly under the roof of fpm, or deferred to the build script mechanism? In Cargo , support of pkg-config-rs, cmake, and other build tools (for C and C++ code) is provided through specific crates. These are then placed in the build-dependencies section:

# Cargo.toml
[build-dependencies]
cmake = "0.1"

Packages which rely on these tools are required to provide a build.rs script, that will typically inspect the system and figure out all the necessary flags, or even download and install missing dependencies. The idea is of course one has a access to a full programming language and not just restrictive manifest entries.

I can see pros and cons to both approaches (pursuing something like pkg-config support directly within fpm vs deferring it to the build script mechanism).

[LKedward]

I can see pros and cons to both approaches (pursuing something like pkg-config support directly within fpm vs deferring it to the build script mechanism).

This is an interesting point; having thought about it, IMO I'd prefer to have direct support within fpm to simplify the user experience as much as possible.

[awvwgk]

We could probably start a separate fpm project to implement a pkg-config wrapper and use it in fpm to separate the two efforts.

[ivan-pi]

Personally, I would support having pkg-config integrated directly (I mean within the manifest, the pkg-config wrapper can be in a separate git project). I have the feeling this could help cover many external dependencies already. But I don't think it is feasible to integrate support for all possible build system files (CMake, meson, scons, etc.) within fpm (at least not in the near future). Also note that pkg-config is not part of the standard MinGW installation, so Fortran users on Windows users could still find themselves with poor support.

@milancurcic, do you have any special reasons why pkg-config was not an option for you with NetCDF support in #438?

[ivan-pi]

We could probably start a separate fpm project to implement a pkg-config wrapper and use it in fpm to separate the two efforts.

Maybe this gives birth to a Fortran implementation of pkg-config 👀. I wonder how much effort that would be (see https://gitlab.freedesktop.org/pkg-config/pkg-config, or https://github.com/pkgconf/pkgconf for C versions).

Addendum: I've scrapped this idea already. An interface to libpkgconf sounds more reasonable. They already support Python bindings this way: https://github.com/pkgconf/pkgconf-py. A feature comparison between the original pkg-config and pkgconf is located at http://pkgconf.org/features.html

Addendum 2: there is also a MIT-licensed Python interface to the command line pkg-config tool. Maybe it is worth looking into for a rough Fortran design.

[milancurcic]

@milancurcic, do you have any special reasons why pkg-config was not an option for you with NetCDF support in #438?

It seems to be a system misconfig. On my Ubuntu 18.10:

$ pkg-config netcdf --cflags
-I/usr/include/hdf5/serial
milan@hyperion:~$ ls /usr/include/hdf5/serial/netcdf.mod
ls: cannot access '/usr/include/hdf5/serial/netcdf.mod': No such file or directory
milan@hyperion:~$ locate netcdf.mod
/home/milan/opt/netcdf-4.6.2_intel19/include/netcdf.mod
/home/milan/opt/netcdf-fortran-4.4.4_intel19/include/netcdf.mod
/usr/include/netcdf.mod

So, pkg-config thinks netcdf.mod is somewhere where it's not. I could do some research about pkg-config and fix the pc file, but now I'm learning another tool. I don't want that.

Setting paths myself works just fine for me. I don't mind people adding pkg-config support into fpm, but it's not for everybody.

[milancurcic]

Back to the original question, there is another approach not mentioned: fpm could search the common system paths itself, such as /usr/include for modules, /usr/lib*/* for libraries etc (Edit: on second thought this may not be needed for linking). I don't think this is a good idea, but putting it out there.

[arjenmarkus]

Well, I experimented with finding and reading the PC files and found it was not all that difficult. At least in this quick-and-dirty implementation. For actual use in fpm it will have to be made more robust, but I tested it in Cygwin with gfortran and in plain Windows with Intel Fortran oneAPI (explicitly setting PKG_CONFIG_PATH) and it produced the value I expected. The code is in the attachment. Op vr 16 apr. 2021 om 18:03 schreef Milan Curcic ***@***.***>:

Back to the original question, there is another approach not mentioned: fpm could search the common system paths itself, such as /usr/include for modules, /usr/lib*/* for libraries etc. I don't think this is a good idea, but putting it out there. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#439 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAN6YR4VFQUFVZRPPN4FQD3TJBNTVANCNFSM42Z75IHQ> .

! pkgconfig.f90 -- ! Limited implementation of a "pkg-config" module ! Limitations: ! - It looks for particular keywords, no translation as pkg-config does ! - On Windows, no examination of the registry ! - It reads the file on every invocation of pkgconfig_get ! module pkgconfig implicit none private public :: pkgconfig_exists, pkgconfig_get contains ! pkgconfig_exists -- ! Check if the required PC file can be found or not ! ! Arguments: ! pkgname Name of the package ! ! Result: ! True if a PC file is found, false otherwise ! logical function pkgconfig_exists( pkgname ) character(len=*), intent(in) :: pkgname integer :: lufile call pkgconfig_open( pkgname, lufile, pkgconfig_exists ) if ( pkgconfig_exists ) then close( lufile ) endif end function pkgconfig_exists ! pkgconfig_open -- ! Open the PC file for reading ! ! Arguments: ! pkgname Name of the package ! lufile LU-number for the opened file ! exists Whether the PC file exists and could be opened or not ! subroutine pkgconfig_open( pkgname, lufile, exists ) character(len=*), intent(in) :: pkgname integer, intent(out) :: lufile logical, intent(out) :: exists character(len=200) :: pcpath, curpath integer :: ierr, k, status exists = .false. call get_environment_variable( 'PKG_CONFIG_PATH', pcpath, status = status ) if ( status == 0 ) then do while ( pcpath /= ' ' ) k = index( pcpath, ':' ) if ( k == 0 ) then k = index( pcpath, ';' ) endif ! ! Windows typically uses "c:" for the drive, very short directory names are unlikely ... ! if ( k > 2 ) then curpath = pcpath(1:k-1) pcpath = pcpath(k+1:) else curpath = pcpath pcpath = ' ' endif open( newunit = lufile, file = trim(curpath) // '/' // trim(pkgname) // '.pc', status = 'old', iostat = ierr ) if ( ierr == 0 ) then exists = .true. exit endif enddo endif end subroutine pkgconfig_open ! pkgconfig_get_raw -- ! Read the value of a variable from the PC file ! ! Arguments: ! lufile LU-number for the opened file ! variable Name of the variable ! value Value as found n the PC file ! subroutine pkgconfig_get_raw( lufile, variable, value ) integer, intent(in) :: lufile character(len=*), intent(in) :: variable character(len=*), intent(out) :: value character(len=200) :: line integer :: ierr, k character(len=1) :: delim rewind( lufile ) value = ' ' do read( lufile, '(a)', iostat = ierr ) line if ( ierr /= 0 ) then exit endif k = index( line, trim(variable) // ':' ) delim = ':' if ( k == 0 ) then k = index( line, trim(variable) // '=' ) delim = '=' endif if ( k == 1 ) then k = index( line, delim ) value = adjustl( line(k+1:) ) exit endif enddo end subroutine pkgconfig_get_raw ! pkgconfig_get -- ! Read the PC file and return the (expanded) value of the requested variable ! ! Arguments: ! pkgname Name of the package ! variable Name of the variable to be found ! ! Result: ! Character string found in the PC file. If the name does not exist, then an empty string is returned ! ! Limitation: ! Maximum length is 200 ! function pkgconfig_get( pkgname, variable ) character(len=*), intent(in) :: pkgname character(len=*), intent(in) :: variable character(len=200) :: pkgconfig_get, value, newvariable integer :: lufile, k, kend logical :: exists pkgconfig_get = ' ' call pkgconfig_open( pkgname, lufile, exists ) if ( .not. exists ) then return endif call pkgconfig_get_raw( lufile, variable, value ) if ( value == ' ' ) then return endif ! ! Is substitution required? ! pkgconfig_get = value do k = index( pkgconfig_get, '${' ) if ( k == 0 ) then exit else kend = index( pkgconfig_get, '}' ) newvariable = pkgconfig_get(k+2:kend-1) call pkgconfig_get_raw( lufile, newvariable, value ) pkgconfig_get = pkgconfig_get(1:k-1) // trim(value) // pkgconfig_get(kend+1:) endif enddo close( lufile ) end function pkgconfig_get end module pkgconfig ! test -- ! Test the above code ! program test_pkgconfig use pkgconfig implicit none character(len=200) :: libs if ( pkgconfig_exists( 'x11' ) ) then write(*,*) 'Found: x11' else write(*,*) 'NOT found: x11' endif libs = pkgconfig_get( 'x11', 'Libs' ) write(*,*) 'LIBS: ', libs end program test_pkgconfig

[awvwgk]

So, pkg-config thinks netcdf.mod is somewhere where it's not. I could do some research about pkg-config and fix the pc file, but now I'm learning another tool. I don't want that.

My guess is that the neither /home/milan/opt/netcdf-4.6.2_intel19/lib*/pkgconfig nor /home/milan/opt/netcdf-fortran-4.4.4_intel19/lib*/pkgconfig are actually present in the PKG_CONFIG_PATH environment variable, but you have a system installation of netcdf which is always found if no custom search path is provided.

This kinda goes back to the original problem, you have an installation, but no mean to tell your system that you want to use it. Installing outside of a package or environment manager leaves this job to the user, which is plain annoying. My personal solution to this problem is a local environment module setup to manage Fortran libraries for all the different compilers I want to use, writing module files is a bit tedious but it served my needs perfectly so far.

Projects like spack and EasyBuild seem to be the next logical step to get a handle on this issue, but those always seemed like overkill for my local development machine.

[milancurcic]

but you have a system installation of netcdf which is always found if no custom search path is provided.

/usr/include/netcdf.mod is not found unless I specify the path to the compiler. I don't think my custom builds under opt/ are relevant here.

[awvwgk]

I think this issue might be related to #441, it is really disappointing to find that GFortran indeed doesn't use any include path by default to search for module files (just checked locally). Additionally, pkg-config is smart enough to drop all include paths and library paths which are already part of the system paths (including values in CPATH and LIBRARY_PATH), unless it is told otherwise with --keep-system-cflags.

That said, it would probably be easier to use a non-system netcdf with GFortran than using a system installation in this setup. Sounds truly broken to me.

[awvwgk]

Let's call it a long-standing bug in GFortran (since 4.3.0): https://gcc.gnu.org/bugzilla/show_bug.cgi?id=35707

[milancurcic]

@awvwgk yes, exactly, and there is a pkg-config misconfig on my system (off the shelf from apt, I didn't touch it). Pkg-config shouldn't think by default that netcdf is in a directory it is not.