Skip to content

Replace custom require loader with process.dlopen and using static path to node.napi.node #71

Open
@CMCDragonkai

Description

@CMCDragonkai

Specification

Native bindings is currently loaded using requireBinding. This is integrated into our native addon packaging style using optional dependencies.

However this won't work as part of ESM. So it has to be changed to using dlopen in preparation for ESM migration.

Furthermore, require supports the package.json specification of main. Which we are currently setting to node.napi.node. However instead we just fix statically where our .node files will be, and that it will always be in @matrixai/quic-linux-x64/node.napi.node.

This means dlopen should work better, and plus it becomes simpler to use when it comes to integrating into esbuild bundling and pkg.

In particular it means:

  • esbuild needs to specify @matrixai/quic-linux-x64 as an external package, since esbuild cannot bundle such a thing - this is done by setting it in optionalDependencies and passing it to the external option in await esbuild.build(esbuildOptions). - The reason it is in optionalDependencies is because this allows it to be installed into the node_modules so when a nix-build occurs, it will in fact exist when esbuild is running. This we cannot really automate, and we have to do this for all native addons. See package.json in Polykey-CLI.
  • pkg needs to specify an asset as node_modules/@matrixai/quic-linux-x64/node.napi.node. - Currently we're using node-gyp-build to auto-find native modules, however this does not work for our custom optional dependency structure. So currently this is also manually specified.

I think it should be possible to update our native module finding code to look for node.napi.node that is within our direct dependencies too, by just iterating through the root directory of each package, then it should be possible to make that automatic as well. That would reduce this:

  "pkg": {
    "assets": [
      "node_modules/@matrixai/quic-linux-x64/node.napi.node",
      "node_modules/@matrixai/quic-linux-x64/package.json",
      "node_modules/@matrixai/quic-darwin-x64/node.napi.node",
      "node_modules/@matrixai/quic-darwin-x64/package.json",
      "node_modules/@matrixai/quic-darwin-arm64/node.napi.node",
      "node_modules/@matrixai/quic-darwin-arm64/package.json",
      "node_modules/@matrixai/quic-win32-x64/node.napi.node",
      "node_modules/@matrixai/quic-win32-x64/package.json",
      "node_modules/@matrixai/mdns-linux-x64/node.napi.node",
      "node_modules/@matrixai/mdns-linux-x64/package.json"
    ],
    "scripts": [
      "dist/polykeyWorker.js"
    ]
  },

Down to:

  "pkg": {
    "assets": [],
    "scripts": [
      "dist/polykeyWorker.js"
    ]
  },

Since it would be auto-found.

Additional context

Tasks

  1. ...
  2. ...
  3. ...

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions