New warning about entry points that might terminate R in compiled code in R-devel

[astamm]

File ‘nloptr/libs/nloptr.so’:
    Found ‘_ZSt4cerr’, possibly from ‘std::cerr’ (C++)
      Object: ‘nlopt/lib/libnlopt.a’
    Found ‘_ZSt4cout’, possibly from ‘std::cout’ (C++)
      Object: ‘nlopt/lib/libnlopt.a’
    Found ‘__printf_chk’, possibly from ‘printf’ (C)
      Object: ‘nlopt/lib/libnlopt.a’
    Found ‘abort’, possibly from ‘abort’ (C)
      Object: ‘nlopt/lib/libnlopt.a’
    Found ‘exit’, possibly from ‘exit’ (C)
      Object: ‘nlopt/lib/libnlopt.a’
    Found ‘putchar’, possibly from ‘putchar’ (C)
      Object: ‘nlopt/lib/libnlopt.a’
    Found ‘puts’, possibly from ‘printf’ (C), ‘puts’ (C)
      Object: ‘nlopt/lib/libnlopt.a’
    Found ‘stderr’, possibly from ‘stderr’ (C)
      Object: ‘nlopt/lib/libnlopt.a’

Compiled code should not call entry points which might terminate R nor
  write to stdout/stderr instead of to the console, nor use Fortran I/O
  nor system RNGs nor [v]sprintf.
  
  See ‘Writing portable packages’ in the ‘Writing R Extensions’ manual.

Taken from the CI: https://github.com/astamm/nloptr/actions/runs/14003511427/job/39276607383?pr=196

[astamm]

Probably related to this bullet

• Some ‘R CMD check’ ‘NOTE’s on the use of non-API entry points have been upgraded to ‘WARNING’s in preparation for removing declarations and, where possible, hiding these entry points.

from https://developer.r-project.org/blosxom.cgi/R-devel/NEWS.

[aadler]

Yes, even so, those are in nlopt and not nloptr. False positive vis-a-vis R?

[astamm]

Indeed, WRE mentions:

Compiled code is checked for symbols corresponding to functions which might terminate R or write to stdout/stderr instead of the console. Note that the latter might give false positives in that the symbols might be pulled in with external libraries and could never be called. Windows58 users should note that the Fortran and C++ runtime libraries are examples of such external libraries.

I cannot find where I read it but indeed, generally speaking,

• either we provide a patch to nlopt sources but that requires maintenance of the patches and might not be adapted here since we also rely on system nlopt if found;
• alert maintainer of upstream external library.

So I will do the latter. However, since it is now a WARNING, I guess we might have to address it so maybe I'll try to submit a PR in nlopt unless the guilty code lines in nlopt are never called by nloptr.

[aadler]

I don't think we should patch the nlopt sources; let them take care of nlopt. Also, they don't need to adhere to CRAN policy, so I don't think alerting them will help. They cannot use the R API, obviously. I think that as long as CRAN accepts that these are false positives and lets us through, we should be OK. If they are really upset about it, then we may have to patch the source, which concerns me.

[aadler]

I've reread WRE, and I think you are right, @astamm. We are going to need to patch the src for 2.10.0 to use R-acceptable APIs for the exits and the warnings. WRE is pretty clear that what we have been doing until now is unacceptable. I mean, we can put the onus on the user to obtain an nlopt library, but that's suboptimal, IMO.

[astamm]

WRE is pretty clear that what we have been doing until now is unacceptable

Can you elaborate on this statement please?

My view is that wrapping an external library written in another language to be usable in R while relying on system install (hence obtained by the user) complicates our maintenance duties as we must be backward compatible to old versions to some extent (the limit of which is unclear) and we indeed simply cannot ask them to abide by the rules CRAN imposes on R developers. So the only way I see is to always compile nlopt from included, possibly patched, sources, using latest available sources.

However, I seem to be the only one with this opinion and others with much more experience than me on porting external libraries to R have strong opposite beliefs. What WRE says also somewhere is that it is recommended to use system libraries if found by pkg-config which might be part of the reason they insist on using system nlopt. And indeed we would be violating this recommendation if we always build nlopt from included patched sources.

[astamm]

I still filed an issue in the nlopt repo (stevengj/nlopt#604) because I noticed that for instance the use of stderr and fprintf is almost exclusively limited to cobyla.c and not other algorithm implementations. So maybe they will be willing to do something about it.

[aadler]

WRE is pretty clear that what we have been doing until now is unacceptable

Can you elaborate on this statement please?

See the following bullets under section 1.6 (emphasis added is my own):

• Under no circumstances should your compiled code ever call abort or exit85: these terminate the user’s R process, quite possibly losing all unsaved work. One usage that could call abort is the assert macro in C or C++ functions, which should never be active in production code. The normal way to ensure that is to define the macro NDEBUG, and R CMD INSTALL does so as part of the compilation flags. Beware of including headers (including from other packages) which could undefine it, now or in future versions. If you wish to use assert during development, you can include -UNDEBUG in PKG_CPPFLAGS or #undef it in your headers or code files. Note that your own src/Makefile or makefiles in sub-directories may also need to define NDEBUG.
  This applies not only to your own code but to any external software you compile in or link to.
  Nor should Fortran code call STOP nor EXIT (a GNU extension).

• Compiled code should not write to stdout or stderr and C++ and Fortran I/O should not be used. As with the previous item such calls may come from external software and may never be called, but package authors are often mistaken about that.

[astamm]

Yes we are on the same page then.

[astamm]

One complication is for Windows users. WRE says:

Many people (including the CRAN package repository) will not accept source packages containing binary files as the latter are a security risk. If you want to distribute a source package which needs external software on Windows or macOS, options include

• To arrange for installation of the package to download the additional software from a URL, as e.g. package Cairo used to.
• To negotiate with Tomas Kalibera to include Windows software in Rtools or with Simon Urbanek to include macOS software in his ‘recipes’ system.
• (For CRAN.) To negotiate with Uwe Ligges to host the additional components on WinBuilder, and write a configure.win file to install them.

They may or may not have curated the sources and they may or may not have used the latest available sources.

For macOS, we could also rely on our included sources but for Windows we rely on Tomas Kalibera Rtools version and should negotiate with him I guess.

[eddelbuettel]

I have not read each message above (and got here via the message to r-sig-win). What you two seem to have missed is that R now checks in static libraries for 'forbidden' symbol such as abort or cout or cerr or (s)printf or ... So you need to patch the embedded libnlopt sources to not use those. One advantage of external shared 'system' libraries is that you avoid this scan for symbols -- so when nloptr built against libnlopt.so (as at CRAN) you were not scanned. When it built locally, you were not scanned because you used static library. Now that changed: the static library is now checked.

And I just did a fix on a similar package of mine (with a small-ish fallback library like libnlopt): I had forgotten to set the needed -DNDEBUG when compiling the fallback embedded static library leading to assert() being compiled in (violates "shall not abort"), and upstream had four uses of sprintf I converted to snprintf to not run afoul of that flagged function; see here for my recent upload post.

Tomas has also very kindly helped convert two package to (in their Windows build) switch to use pkg-config (where available) and thus pivot to Rtools provided builds. One was RQuantLib (and QuantLib is unpleasant to build but he has it), the other needed GSL which he has too. So yes you may want to talk to him about a) whether he has NLopt built and b) help with src/Makevars.win.

[astamm]

Thanks Dirk.

We did not miss that R now checks in static libraries and that we have to patch the included sources. This thread is exactly about that.

What we missed though is the fact that R does not check for forbidden symbols when external shared system libs are used. Which is the reason why R CRAN checks are all in OK status I guess.

But the fact that R does not check for forbidden symbols when external shared system libs are used does not mean these libs do not use those symbols, and R could terminate unexpectedly.

So I guess we could reason short-term and do nothing because CRAN is happy or only patch the current sources to satisfy our CI.

But in the long run, NLopt is using symbols forbidden by CRAN independently from the type of lib that is built (static or shared), which might terminate R in unexpected ways. It is likely that NLopt maintainers will not deal with CRAN rules because they are not the ones providing NLopt to the R community. So we were even considering to provide patched sources (which anyway we'll have to do) and always use NLopt compiled from these sources, even if a system nlopt exists. This violates the CRAN rule of using system lib if detected by pkg-config but, if we do not do that, we violate another CRAN rule by using forbidden symbols in external sources. I tend to believe that it is better to prevent R from crashing unexpectedly for downstream users. In other words, we would have nloptr completely self-contained, which is what many packages also do.

But you have way more experience that I do in providing external software to the R community and you are not in favour of this so there must be something I do not yet understand.

[eddelbuettel]

Sorry if I missed a part of the discussion, I came here late as I said. I do not understand what your one-but-last paragraph is trying to say. But I concur with what Tomas replied to Avi on the r-sig-windows list: this belongs to r-package-devel, not here or on a Windows only list. As with other aspects of packages interfacing and/or bundling external libraries, you do not have to sort this out alone as it has been done by many other packages too.

[astamm]

Is there a link to the conversation on r-sig-windows please?