official aws lambda layer for @duckdb/node-api

[cookiejest]

Hey, is there an official aws lambda layer for @duckdb/node-api or we must include as part of our own deployment pipeline?

Currently getting this error on a clean lambda function. I guess most people would also get this:

   "errorType": "Runtime.ImportModuleError",
    "errorMessage": "Error: Cannot find module '@duckdb/node-bindings-linux-x64/duckdb.node'\nRequire stack:\n- /var/task/index.mjs",
    "stack": [
        "Runtime.ImportModuleError: Error: Cannot find module '@duckdb/node-bindings-linux-x64/duckdb.node'",

Many thanks!

[jraymakers]

There is not. The lambda layer for the old Node client isn't "official", either; it's provided by a member of the community.

I'm not very familiar with lambda layers so I don't know what's involved. I don't understand enough about the lambda environment to know why you got the error above, but my guess is there's something different about the Node version or configuration.

I'd welcome any concrete suggestions or PRs to make this work. I have a fair amount of other items on my list right now, so I'm not likely to have time to look into this anytime soon.

[cookiejest]

figured it out actually. It was due to lambda function running on a different chip to my local machine and the .node binding dependency not being installed when the function was deployed on aws.

I wouldn't want to use a random lambda layer as abit of a security risk but these are the steps i did to get it working (for others wanting to fix the same issue):

1. Confirm what lambda run time your using (if nodejs its probably linux64)
2. Clone your node_modules/@duckdb directory locally and delete the large files you dont need as it needs be below 250mb uncompressed and under 50mb compressed to be uploadable as a lambda layer. (I deleted duckdb-wasm and the binding that was for my local arm machine).
3. Get the correct binding for the environment the lambda function is running on (can be forced like this):
   npm i @duckdb/node-bindings-linux-x64 --force
   Here was mine:

5. zip it up and upload as a lambda layer and assign it to the lambda function

Now duckdb will be executable inside a lambda function without using a custom docker image.

[jraymakers]

Nice!

I can confirm that those three packages (@duckdb/node-api, @duckdb/node-bindings, and @duckdb/node-bindings-linux-x64) should be the minimum needed (for Linux x64).

[jraymakers]

Probably worth documenting this and adding it to the README. I'll keep this issue open to track that.

[cookiejest]

good idea! I think this would allow alot more people who are just using front end frameworks like aws amplify to use duckdb in their lambda functions :)

[hannesj]

When bundling a lambda function with SST v2, the installation of the native binary can be can be configured as follows:

  new Function(stack, "InvoiceFunction", {
    handler: "functions/invoice.handler",
    nodejs: {
      esbuild: {
        loader: { ".node": "file" },
      },
      install: ["@duckdb/node-bindings-linux-arm64"],
    },
  });

[cookiejest]

That's much better !🤗 (if your using sst)

[subos2008]

What's the use case for DuckDB in lambdas? Given they throw their memory and filesystem away after each call. You could mount a filesystem into the lambda but that wouldn't be safe unless invocation concurrency was hard coded to 1.

Asking because I'm porting from DuckDB to DynamoDB for AWS deployment into lambdas myself as I couldn't see a way to use DuckDB in lambdas

[cookiejest]

The primary use is if your data is already in S3 or s3 tables and you have a complex analytical query your running on every request you don't want a separate database for (or your trying to reduce athena use), you can use the /tmp folder to go out of memory if you really need too.

Using dynamodb for analytical queries would be a world of pain. You would be better with aurora but then there is more to manage and has its own problems.