Proposal: Method to build NIFs using the Gleam build tool

[Enderchief]

Proposal: Method to build NIFs using the Gleam build tool

Problem

Compiling NIFs for Gleam

Distributing a Gleam package that includes a NIF (Native Implemented Function) is not possible with the current Gleam build tool without distributing the precompiled shared libraries (SO/DLL) for every platform and choosing the correct one at runtime. This could lead to unnecessarily large packages due to the amount of outputs needed.
To build a C NIF from source, a c compiler has to be called from the command line. An example command using GCC would be gcc -o "./priv/add_nif.so" -fpic -shared c_src/add_nif.c c_src/add.c (This command runs on Linux with GCC and MacOS with a symbolic link to LLVM C Compiler). On Windows, it would be cl -LD -MD -Fe "priv\add_nif.dll" "c_src\add_nif.c" "c_src\add.c".

Current Solutions

Distributing all shared libraries in a package

Instead of compiling at Gleam build time, compile the NIF for each target in advance and include that in the package. Although it seems simple, it would mean having to compile at the minimum for: Linux x86, Linux ARM64, MacOS x86, MacOS ARM and Windows x86 (or more for 64-bit platforms, *BSD, etc.). Additionally, this would substantially increase the size of a project with unnecessary files.

Use an alternative build tool

Using Rebar3 (most common) or Erlang.mk. These allow providing commands to run before building the package. The problem with this is losing the standard Gleam tool.

Build during execution

When running Gleam, the package checks if the shared library has been compiled and if not, starts a subprocess to compile it.

Make the package user build the library

Either give instructions or provide a command (ex: gleam run -m foo/build) which builds during its execution. This would require explicitly building the package by a user and in some cases can be left unnoticed with a user wondering the cause of an error.

Goals

Not all required but it would be nice to have all.

• Build C NIFs for cross-platform (including Windows)
• Linking to other shared libraries (-lNAME with gcc)
• Specifying multiple files to build
• Conditional building based on platform (ex: using cl instead of gcc if the platform is Windows)
• Prevent doing hacky things

Try to follow this quote

Not super keen on hooks [...] because people use them for all sorts of gnarly stuff
[...] an API that doesn't encourage that sort of thing
- lpil

Other Languages and Systems

As a basis, provided is a list of how other tools/language would handle building a C library for FFI.

Erlang (Rebar3)

Pre-build and post-build hooks specified in the rebar.config

{pre_hooks,
  [{"(linux|darwin|solaris)", compile, "make -C c_src"},
   {"(freebsd)", compile, "gmake -C c_src"}]}.
{post_hooks,
  [{"(linux|darwin|solaris)", clean, "make -C c_src clean"},
   {"(freebsd)", clean, "gmake -C c_src clean"}]}.

Erlang (Erlang.mk)

Either evokes the default make target if a Makefile is detected or will be built when environment variables such as C_SRC_DIR are defined.

Elixir (Mix)

The compilers property in mix.exs specifies functions that should be run on compiling with Mix.

defmodule Mix.Tasks.Compile.MyNifs do
  def run(_args) do
    {result, _errcode} = System.cmd("make", [], stdout_to_stderr: true)
    IO.binwrite(result)
  end
end

...

defmodule MyNifs.Mixfile do
  use Mix.Project

  def project do
    [app: :my_nifs,
     compilers: [:my_nifs] ++ Mix.compilers,
     ...]
  end
end

more info

Python (setuptools)

Specify library name and list of files in setup.py (the standard for building package for Python) and set environment variables for options to the C compiler.

# setup.py
from setuptools import Extension, setup

setup(
    ext_modules=[
        Extension(
            name="mylib.foo",`
            sources=["foo.c", "bar.c"],
        ),
    ]
)

more info

Node.js (node-gyp)

Specify files and compiler options in a binding.gypi file. (alternatively, the Node.js ecosystem prefers building in advance and installing the correct binary post install)

{
  "targets": [
    {
      "target_name": "main",
      "sources": [ "main.c" ],
      "include_dirs": [
        "build"
      ]
    }
  ]
}

Go (cgo)

Specify compiler flags as comment directives at the root of a package.

// #cgo CFLAGS: -DPNG_DEBUG=1
// #cgo amd64 386 CFLAGS: -DX86=1
// #cgo LDFLAGS: -lpng
// #include <png.h>
import "C"

Java (Maven + maven-compiler-plugin)

Specify compiler options in the pom.xml file.

<project>
    [...]
    <build>
        [...]
        <plugins>
            <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.13.0</version>
            <configuration>
                <sources>
                    <source>
                        <directory>${basedir}/src/main/cpp/jni</directory>
                        <fileNames>
                            <fileName>main.cpp</fileName>
                            <fileName>foo.c</fileName>
                        </fileNames>
                    </source>
                </sources>
                <compilerExecutable>g++</compilerExecutable>
                <compilerArgs>
                <arg>-verbose</arg>
                <arg>-Xlint:all,-options,-path</arg>
                </compilerArgs>
            </configuration>
            </plugin>
        </plugins>
        [...]
    </build>
    [...]
</project>

Options to Implement

Gleam invokes make if a Makefile exist (Primarily Suggested)

Many Erlang packages that have a NIF, build under the assumption the machine has make installed. For Linux and MacOS developers, most have make and gcc on their system. For Windows developers, many use MinGW/Mingw-w64 which provide these tools.

In gleam.toml an option is provided whether to call make or not on build. When true, Gleam during build will run make. Library developers can choose what to put within their Makefile.

name = "hi"
version = "1.0.0"

# by default, it is false. If a Makefile is detected, it is true unless specified false.
nif = false 

[dependencies]
gleam_stdlib = ">= 0.34.0 and < 2.0.0"

Gleam Runs the Compiler, You give the Options. (Alternative Suggestion)

In gleam.toml

name = "my_package"
version = "1.0.0"

[build]
name = "add"                      # name of the output. `NAME_nif.{so|dll}`
sources = ["c_src/add.c", "c_src/add_nif.c"]    # source files to compile
include_dirs = ["./include"]      # directories to search for headers
link = ["somelibrary"]            # libraries to link to
# executable = "g++"              # defaults to gcc
# additional_args = ["list", "of", "argument"]

# optionally specify different values for platforms
[build.windows] # this corresponds to [build.$(uname)]
sources = ["c_src/add.c", "c_src/add_nif.c", "c_src/win.c"]


[dependencies]
gleam_stdlib = ">= 0.34.0 and < 2.0.0"

In non-Windows platforms this will run as gcc -o "./priv/add_nif.so" -fpic -shared -I./include -lsomelibrary c_src/add.c c_src/add_nif.c

In Windows, Gleam will search for first GCC. If found, it will use the same command as above, replacing the POSIX paths to Windows paths (gcc -o ".\priv\add_nif.dll" -fpic -shared -I.\include -lsomelibrary c_src\add.c c_src\add_nif.c c_src\win.c).
If not, it will throw a build error.

If not, it will fall back to using cl with the command cl /LD /MD /Fe /I.\include "priv\add_nif.dll" "c_src\add_nif.c" "c_src\add.c" "c_src\win.c" (forcing the use of gcc so that there is a common API and maintaining a build system that also uses MSVC would be tedious)

The only downside is inablilty to compile a non-C/C++ library. But doing so does require more work normally (although currently Rustler and zigler exist which make it easier). The upside is forcing people to go through one method leads to discouragement of "gnarly stuff".

Further Notes

Alternatively, I had considered (and started but stopped) writing about having pre/post build hooks (similar to Rebar3) but decided against it because it makes the method to building a NIF too open-ended rather than having one thing and keeping things consistent.
Looking at NIF based packages on Hex, I discovered many require Make and GCC, so it would be reasonable to assume someone using a NIF has those on their system.
Something important was to remember Windows developers. According to the Gleam 2022 Developer Survey, 14.15% of systems used are Windows for development (out of 318). As Gleam has grown so much in the past two months to people not normally familiar with BEAM, there are many more developers to consider.
I would like to thank the Gleam community for giving their requirements which I had searched through on Discord, also thanking the people behind all the tools I mentioned which served as an inspiration.

[lpil]

I'm used to all of the referenced prior-art being the most painful parts of those ecosystems, either due to unreliable build scripts or due to them being abused to do other things.
They all work pretty much the same way. Are there any tools that have a different design?

For the makefile approach is that suitably cross-platform? I don't know how popular and easy to use make is on Windows. Is it something Windows users have installed?

How would Windows compilation work for the version where we run the C compiler directly?

How would the non-make version work for compiling native deps beyond very simple C code, where some other compiler or more sophisticated build tool is needed.

If we have make support it will need to not be automatically run if present. It needs to be opt in and directed in some fashion.

[Enderchief]

For the makefile approach is that suitably cross-platform? I don't know how popular and easy to use make is on Windows. Is it something Windows users have installed?

By default (i.e. developers who have basic C/C++ build tools), there is something similar called nmake although not compatible with make. An alternative is telling people who would like to use NIFs to make sure they have Mingw-w64 which is very popular as creates support for the GCC compiler (along with tools like Make) to run on Windows. Most of the alternatives are, the developer writes a build file and some build tool (which everyone has to install) gets run.

How would Windows compilation work for the version where we run the C compiler directly?

Either require Windows users to have GCC (from Mingw-w64), or using the cl command on Windows to build. It would just mean that Gleam has to run a separate command for *NIX and Windows. gcc -o "./priv/add_nif.so" -fpic -shared -I./include c_src/add.c c_src/add_nif.c and cl /LD /MD /Fe /I.\include "priv\add_nif.dll" "c_src\add_nif.c" "c_src\add.c" "c_src\win.c".

How would the non-make version work for compiling native deps beyond very simple C code, where some other compiler or more sophisticated build tool is needed.

It would not work but maybe that is for the best. I looked into Erlang projects with NIFs (searched for NIF on Hex and GitHub) and found most used either CC/GCC via a Makefile or from rebar.config. Having a NIF of another language such as Rust is a lot more complicated in my limited experience without a helper library such as Rustler.

[ericphanson]

They all work pretty much the same way. Are there any tools that have a different design?

FWIW Julia (https://julialang.org/) uses a pretty different design: binaries are cross-compiled ahead of time, then the compiled artifacts are distributed with auto-generated wrapper packages that are managed by the ordinary package manager.

• binary builder tool to cross-compile libraries: https://binarybuilder.org/
• community build tree: https://github.com/JuliaPackaging/Yggdrasil
• PackagingCon talk: https://www.youtube.com/watch?v=S__x3K31qnE

The end result is very smooth for the end-user (eg Pkg.add("CUDA") installs a full GPU stack relatively quickly, reproducibly, and without compilation), but it requires a bit more upfront work from the package author, and the community build tree often involves some very hard working volunteers helping folks get random libraries cross-compiling.

[lpil]

This makes auditing challenging, so you'd need to also provide some sort of audited verification system alongside the binaries for this approach to be usable in enterprise where they have security obligations. This is out-of-reach for packages that do not have a company behind them willing to make this level of financial investment and tolerate this level of compliance hoop-jumping.

[ericphanson]

The build scripts are open source, the binaries come with hashes and the build environment attempts to make builds reproducible (setting timestamps and such), but I don’t know if that’s enough. I know Julia is used in some security critical environments but I don’t know how binaries are handled in those environments. (Julia is also more a computational language so I think it is run disconnected from the internet in some security critical environments, which is probably infeasible for Gleam). There is also an overrides system so locally built or system binaries can be substituted with configuration changes instead of code changes.

[lpil]

It's not so much a technical problem, it's more that the organisation that provides the binaries then needs to then get the appropriate security and compliance certifications, which is an expensive and time consuming ongoing process.

[CrowdHailer]

They all work pretty much the same way. Are there any tools that have a different design?

I think dagger.io is a tool that has a slightly different approach and is tackling some of these issues. i.e. reproducabile cross platform builds. I'm not yet sure how it handles cross platform. I'm experimenting to use it as a way to standardise all my CI and deploy scripts.

[SichangHe]

I've heard claims that zig build works truly cross-platform without the need of third party tools. Not sure how true that is and how they do it, though.

[Enderchief]

Another alternative is allowing the creation a build script written in Gleam. This would come with a new package to facilitate the compilation.
(Inspired by Rust and Zig.)

Assuming the following file structure:

hello
├── c_src
│ ├── add.c
│ ├── add_nif.c
│ └── add.h
├── src
│ └── hello.gleam
├── test
│ └── hello_test.gleam
├── build.gleam
├── gleam.toml
└── manifest.toml

//// build.gleam

import gleam/build as b

pub fn main() {
  let assert Ok(_) =
    b.new(name: "add_nif")
    |> b.add_source("c_src/add_nif.c")
    |> b.add_header("c_src/add.h")
    |> b.add_library("libwobble")
    |> b.build()
}

Before gleam build compiles the author's regular Gleam package, it runs the build.gleam file. Afterwards, the file structure looks like the following
hello
├── c_src
│ ├── add.c
│ ├── add_nif.c
│ └── add.h
├── priv
│ ├── add_nif.so (or .dll on Windows)
├── src
│ └── hello.gleam
├── test
│ └── hello_test.gleam
├── build.gleam
├── gleam.toml
└── manifest.toml

---

I have created an example just to showcase the idea. https://github.com/Enderchief/gleam_nif_example/blob/master/hello/test/build.gleam

[Enderchief]

Instead of relying on a developer to have GCC (or any compiler) installed, Gleam could instead embed Tiny C Compiler which is ~100KB.
Bun (JavaScript runtime) does that for their C FFI.

[lpil]

It cannot be TCC as the performance of the generated code is poor and would have an impact on these Gleam apps in production.