Add Esbuild support

[bretambrose]

Describe the feature

Using esbuild to build/package a JS project that uses the CRT is an exercise in frustration. We should smooth the sharp corners and provide README and/or sample-based guidance for people who wish to use esbuild.

Ideally we should provide solutions that

1. use the browser distribution
2. use the node distribution and support flexible lookups for the NAPI module, with detailed instructions

We may want to special case Lambda as well with a targeted sample.

We've also had a request for flexible NAPI module path lookup here: #214

I believe proper esbuild support would have to include flexible NAPI module resolution, and so I'm consolidating that request into this feature request as well.

Use Case

N/A

Proposed Solution

No response

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

[Lewenhaupt]

@bretambrose Any update on this? I was facing an issue with aws-iot-device-sdk and tried upgrading to v2 but I fail to figure out how I can make aws-crt avialble in the lambda. Both when bundle with esbuild but also when deploying it in a layer.
Any guidance would be greatly appreciated.

[bretambrose]

Don't have an update. It's in the backlog, but hasn't been prioritized/scheduled.

[mitgol]

The problem is in the code for the aws-crt package. In the node_modules/aws-crt/dist/native/binding.js file, there is this:

const binary_name = 'aws-crt-nodejs';
const platformDir = `${os_1.platform}-${os_1.arch}-${cRuntime}`;
let source_root = path.resolve(__dirname, '..', '..');
const dist = path.join(source_root, 'dist');
if ((0, fs_1.existsSync)(dist)) {
    source_root = dist;
}
const bin_path = path.resolve(source_root, 'bin');
const search_paths = [
    path.join(bin_path, platformDir, binary_name),
];
let binding;
for (const path of search_paths) {
    if ((0, fs_1.existsSync)(path + '.node')) {
        binding = require(path);
        break;
    }
}
if (binding == undefined) {
    throw new Error("AWS CRT binary not present in any of the following locations:\n\t" + search_paths.join('\n\t'));
}

As you can see, this is trying to load some binary modules. It is making the assumption that the code that is running is two directories down from the project root, and that there will be a binary in the dist/bin directory. That works fine when running on an EC2, but not when it’s packaged into a single file. When it runs, it yields this error:

{
  "errorType": "Error",
  "errorMessage": "AWS CRT binary not present in any of the following locations:\n\t/bin/linux-x64-glibc/aws-crt-nodejs",
  "trace": [
    "Error: AWS CRT binary not present in any of the following locations:",
    "\t/bin/linux-x64-glibc/aws-crt-nodejs",
    "    at node_modules/aws-crt/dist/native/binding.js (/var/task/index.js:157153:13)",
…
  ]
}

Clearly, it’s not going to find the binaries in /bin ! In fact, as far as I can tell, the binaries haven’t been copied anywhere.
To make it work, there are two steps. (1) add the binaries to the zip, and (2) Make sure there is a dot in front of the path.
When packaged by esbuild, it looks like this:

    var binding;
    for (const path2 of search_paths) {
      if ((0, fs_1.existsSync)(path2 + ".node")) {
        binding = require(path2);
        break;
      }
    }

It’s possible to fix it by making a change to one line:

    var binding;
    for (const path2x of search_paths) { const path2 = '.' + path2x;
      if ((0, fs_1.existsSync)(path2 + ".node")) {
        binding = require(path2);
        break;
      }
    }

I have the below in my package.json file to make sure this happens:

"clean": "rm -rf build/dist build/index.zip node_modules",

"build": "esbuild src/index.ts --bundle --sourcemap --platform=node --target=es2022 --outfile=build/dist/index.js && cp -r configuration/aws_lambda build",

"postbuild": "cp -r node_modules/aws-crt/dist/bin build/dist && cd build/dist && chmod -R ugo+rx bin && sed -i.bak \"s/^    for (const path2 of search_paths) {$/    for (const path2x of search_paths) { const path2 = '.' + path2x;/\" index.js && rm -f ../index.zip index.js.bak && zip -r ../index.zip .",

I should point out that I have not gotten it to work with webpack, because webpack is more intrusive in what it does to the code. Instead of using require, it has something called __webpack_require__ which is more complicated, and the simple hack above still doesn’t enable finding the binaries. Someone who understands it better (or at least has the time and inclination) should probably be able to figure out how to get that to work.

Here’s a link talking about webpack_require which may be of interest: https://devtools.tech/blog/understanding-webpacks-require---rid---7VvMusDzMPVh17YyHdyL

[alonw-cf]

@bretambrose we encounter the same issue here, where we use TypeScipt+esbuild for lambda functions. Any change to increase priority?

[justin-masse]

Having the same exact issue here, unable to load the binaries in lambda when using esbuild and compiling it all into one single file.

This is a major blocker for supporting multi-region services behind a cloudfront distribution. CloudFront cannot guarantee that you will stay within the region that the client request originated from. It is possible that CF directs someone in Virginia to Salt Lake City which would fall into us-west buckets instead of us-east. This means someone would be signing a sigv4 request expecting us-east-1 and then hits us-west-2 (thanks cloudfront) and the signature is not valid. Thus, we need sigv4a to work in lambda to support this * region.

[mitgol]

Justin - I am pretty sure that eslint just doesn't include the binaries in the giant zip file it makes. If you follow the changes I outlined on Dec 12 you should be able to put it together. Note the "cp -r node_modules/aws-crt/dist/bin build/dist" in the postbuild step which adds the binaries to the zip.

Having the same exact issue here, unable to load the binaries in lambda when using esbuild and compiling it all into one single file.

[justin-masse]

Justin - I am pretty sure that eslint just doesn't include the binaries in the giant zip file it makes. If you follow the changes I outlined on Dec 12 you should be able to put it together. Note the "cp -r node_modules/aws-crt/dist/bin build/dist" in the postbuild step which adds the binaries to the zip.

Having the same exact issue here, unable to load the binaries in lambda when using esbuild and compiling it all into one single file.

So I included aws-crt as a layer instead so it would have the files necessary but it still doesn't seem to work even when referencing the files from the layer. I'm not sure why it won't work as a layer unless the compiled index.js references are still broken even when using them?

[anisg]

Thank you @mitgol, for your assistance.

I encountered the same issue with my Node.js Lambda (ARM64) when trying to use a package that use aws-crt.

Fortunately, I managed to resolve it with following bash commands:

# generate folder "build" with index.js inside
esbuild --bundle --platform=node --target=node16 --minify --loader:.html=text --outfile=build/index.js lambda/lambda.ts 

# add binary "aws-crt-nodejs.node" to the folder "build", inside subfolder "build/bin/linux-arm64-glibc"
mkdir -p build/bin/linux-arm64-glibc && cp ./node_modules/aws-crt/dist/bin/linux-arm64/aws-crt-nodejs.node build/bin/linux-arm64-glibc/.

# fix the index.js output so it looks for the binary locally
node ./scripts/fix-aws-crt-not-found.js ./build/index.js 

# zipping the build folder and uploading it to lambda
rm -f ./build.zip && zip -j ./build.zip ./build/index.js && cd ./build && zip -r ../build.zip ./bin
aws lambda update-function-code --function-name YOUR_LAMBDA_NAME --zip-file fileb://build.zip --region YOUR_REGION --profile YOUR_PROFILE

with scripts/fix-aws-crt-not-found.js looking like this: (It adds a prefix "." to the search path for the binary, just as @mitgol suggested.)

// scripts/fix-aws-crt-not-found.js

const fget = (filepath) => require("fs").readFileSync(filepath).toString()
const fput = (filepath, data) => require("fs").writeFileSync(filepath, data)

const filename = process.argv[2]
const out = fget(filename).replace(
  /=\[(\w+)\.join(.*?)\+"\.node"\]/g,
  '=["."+$1.join$2+".node"]'
)
fput(filename, out)

Hope this helps a bit

[ori-gold-px]

Same exact issue when using ESBuild to compile for Lambda@Edge.

I tried the solution offered by @anisg (not minified, but same idea), and for some reason this gave me the Please check whether you have installed the "@aws-sdk/signature-v4-crt" package explicitly error even though I definitely did import the package.

The error was being thrown in the snippet below because, for reasons unknown to me, CrtSignerV4 was still null:

let CrtSignerV4 = null;
try {
  CrtSignerV4 = signatureV4CrtContainer.CrtSignerV4;
  if (typeof CrtSignerV4 !== "function")
    throw new Error();
} catch (e) {
  e.message = `${e.message}
Please check whether you have installed the "@aws-sdk/signature-v4-crt" package explicitly. 
You must also register the package by calling [require("@aws-sdk/signature-v4-crt");] or an ESM equivalent such as [import "@aws-sdk/signature-v4-crt";]. 
For more information please go to https://github.com/aws/aws-sdk-js-v3#functionality-requiring-aws-common-runtime-crt`;
  throw e;
}

I did something very, very ugly to work around it.

// initializing variable in the global scope
let GlobalCrtSignerV4;

// ----

// setting the global variable wherever the CrtSignerV4 is defined
__name(getHeadersUnsignable, "getHeadersUnsignable");
GlobalCrtSignerV4 = CrtSignerV4; // added this line
import_signature_v4_multi_region.signatureV4CrtContainer.CrtSignerV4 = CrtSignerV4;
import_util_user_agent_node.crtAvailability.isCrtAvailable = true;

// ----

// using the global variable in the try/catch
let CrtSignerV4 = null;
try {
  CrtSignerV4 = GlobalCrtSignerV4; // modified this line
  if (typeof CrtSignerV4 !== "function")
    throw new Error();
} catch (e) {
  // ...
}

Hoping there's a better way to do all this real soon.

[grant-d]

@mitgol thank you so much, this saved me a lot of time and pain.

Hopefully there will be a real solution soon.

[mitgol]

@grant-d This problem is sort-of fixed in the latest release of aws-crt (1.21.1). An environment variable was added that you can set to add a specific path for the binary. The environment variable needs to be set in the Lambda configuration. In CDK the lambda is deployed like this:

const lambda = new Function(this, 'lambdaName', {
  code: ... // Wherever the code comes from
  handler: 'index.handler',
  description: ...,
  functionName: `NameOfTheLambda`,
  environment: {
    AWS_CRT_NODEJS_BINARY_RELATIVE_PATH: 'bin/linux-x64-glibc/aws-crt-nodejs.node',
  },
  runtime: Runtime.NODEJS_18_X,
  ...
});

The new section is environment. You can also set the environment variable manually in the AWS Console under Configuration > Environment Variables.
You can then package the code like this (in package.json)

"build": "esbuild src/index.ts --bundle --sourcemap --platform=node --target=es2022 --outfile=build/dist/index.js && cp -r configuration/aws_lambda build",
"postbuild": "mkdir -p build/dist/bin/linux-x64-glibc && cp node_modules/aws-crt/dist/bin/linux-x64-glibc/aws-crt-nodejs.node build/dist/bin/linux-x64-glibc && cd build/dist && chmod -R ugo+rx bin && rm -f ../index.zip && zip -r ../index.zip .",

Note that this time around the packaging does not include all the other architectures, since only one is specified anyway in the relative path env variable.

[justin-masse]

I know there’s a workaround but this is still a nightmare to implement a change in our builds for 50+ services to support sigv4a in our auth service. I don’t want to have to bundle the binaries manually after I run esbuild.

The workaround of including it as a layer as well is a pain as every single lambda has to update and have that there before we toggle sigv4a on in auth service

[mitgol]

I agree it's sort of clunky, and it's annoying to have to change the deployment of the Lambda to include the environment variable. The one comment I would make is that if you change the package.json as I listed in the comment on Apr 10, there is nothing to do manually. The postbuild step copies the binary into the lambda's zip file.

[justin-masse]

I agree it's sort of clunky, and it's annoying to have to change the deployment of the Lambda to include the environment variable. The one comment I would make is that if you change the package.json as I listed in the comment on Apr 10, there is nothing to do manually. The postbuild step copies the binary into the lambda's zip file.

Yeah unfortunately requires us to update a ton of services which is annoying. This sigv4a logic is abstracted within a sub package we use so our services have no direct knowledge of this code and putting that build logic into dozens of services seems bad.

Honestly just shocked no support yet. Would be cool if this was natively bundled into the lambda runtime even so I could always count on it being present but I know that's not feasible.

[perpil]

@mitgol Thanks for the pointers! I was able to get it to work with the CDK and NodeJsFunction like this:

new NodejsFunction(
      this,
      "Function",
      {
        //<snip>
        environment: {
          AWS_CRT_NODEJS_BINARY_RELATIVE_PATH:
            "bin/linux-arm64-glibc/aws-crt-nodejs.node",
        },
        bundling: {
          //<snip>
          commandHooks: {
            beforeBundling(inputDir: string, outputDir: string): string[] {
              return [
                `mkdir -p ${outputDir}/bin/linux-arm64-glibc && cp node_modules/aws-crt/dist/bin/linux-arm64-glibc/aws-crt-nodejs.node ${outputDir}/bin/linux-arm64-glibc`,
              ];
            },
            afterBundling(inputDir: string, outputDir: string): string[] {
              return [];
            },
            beforeInstall(inputDir: string, outputDir: string): string[] {
              return [];
            },
          },
        },
      }
    );

I'm using arm64 so replace arm64 with x64 if you're using x64