official aws lambda layer for @duckdb/node-api

[cookiejest]

Hey, is there an official aws lambda layer for @duckdb/node-api or we must include as part of our own deployment pipeline?

Currently getting this error on a clean lambda function. I guess most people would also get this:

   "errorType": "Runtime.ImportModuleError",
    "errorMessage": "Error: Cannot find module '@duckdb/node-bindings-linux-x64/duckdb.node'\nRequire stack:\n- /var/task/index.mjs",
    "stack": [
        "Runtime.ImportModuleError: Error: Cannot find module '@duckdb/node-bindings-linux-x64/duckdb.node'",

Many thanks!

[jraymakers]

There is not. The lambda layer for the old Node client isn't "official", either; it's provided by a member of the community.

I'm not very familiar with lambda layers so I don't know what's involved. I don't understand enough about the lambda environment to know why you got the error above, but my guess is there's something different about the Node version or configuration.

I'd welcome any concrete suggestions or PRs to make this work. I have a fair amount of other items on my list right now, so I'm not likely to have time to look into this anytime soon.

[cookiejest]

figured it out actually. It was due to lambda function running on a different chip to my local machine and the .node binding dependency not being installed when the function was deployed on aws.

I wouldn't want to use a random lambda layer as abit of a security risk but these are the steps i did to get it working (for others wanting to fix the same issue):

1. Confirm what lambda run time your using (if nodejs its probably linux64)
2. Clone your node_modules/@duckdb directory locally and delete the large files you dont need as it needs be below 250mb uncompressed and under 50mb compressed to be uploadable as a lambda layer. (I deleted duckdb-wasm and the binding that was for my local arm machine).
3. Get the correct binding for the environment the lambda function is running on (can be forced like this):
   npm i @duckdb/node-bindings-linux-x64 --force
   Here was mine:

5. zip it up and upload as a lambda layer and assign it to the lambda function

Now duckdb will be executable inside a lambda function without using a custom docker image.

[jraymakers]

Nice!

I can confirm that those three packages (@duckdb/node-api, @duckdb/node-bindings, and @duckdb/node-bindings-linux-x64) should be the minimum needed (for Linux x64).

[jraymakers]

Probably worth documenting this and adding it to the README. I'll keep this issue open to track that.

[cookiejest]

good idea! I think this would allow alot more people who are just using front end frameworks like aws amplify to use duckdb in their lambda functions :)