node-fetch request: HPE_INVALID_HEADER_TOKEN

[b3rimbau]

My 11ty application uses node-fetch to pull in some 3rd party data. Unfortunately the service that provides this data is breaking the HTTP protocol and the request is resulting in the error described in this Stackoverflow thread. The actual FetchError from 11ty is:

FetchError: request to [resource URL here] failed, reason: Parse Error: Invalid header value char
    at ClientRequest.<anonymous> ([path to 11ty app]\node_modules\node-fetch\lib\index.js:1491:11)
    at ClientRequest.emit (events.js:400:28)
    at TLSSocket.socketOnData (_http_client.js:522:9)
    at TLSSocket.emit (events.js:400:28)
    at addChunk (internal/streams/readable.js:290:12)
    at readableAddChunk (internal/streams/readable.js:265:9)
    at TLSSocket.Readable.push (internal/streams/readable.js:204:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23) {
  type: 'system',
  errno: 'HPE_INVALID_HEADER_TOKEN',
  code: 'HPE_INVALID_HEADER_TOKEN'
}

The Stackoverflow thread suggests two answers:

1. invoke Node with the --insecure-http-parser option
2. "monkey patch" a different parser.

I am using node-fetch v2.6.7 because I don't think I can use v3 yet because it is an ES Module. Whereas node-fetch v3 supports the insecure-http-parser option, I don't think v2 does.

I lack the experience to know if either of these is a viable option within 11ty and wondered if anyone here could offer some insight?

Many thanks
Chris

[pdehaan]

@b3rimbau Can you share which third party API is returning invalid headers?
If I can reproduce the error locally I can help you try and see if those Stackoverflow workarounds will work or if we need to try and come up with something a bit different.

[b3rimbau]

@pdehaan I've stripped out the API key because I'm not permitted to share it, so you'll not get the JSON data (you'll get a HTTP 401 unauthorized), but for the purpose of recreating the invalid header I think this works:

https://gateway.euronext.com/api/instrumentDetail

Thanks for taking a look.

[pdehaan]

Hey, sorry for the slow reply.

So far I've been unable to get this to work w/ node-fetch@2, axios, or got@11 (or using the --insecure-http-parser option).

const axios = require("axios");
const got = require("got");
const fetch = require("node-fetch");

const URL = "https://gateway.euronext.com/api/instrumentDetail";
// const URL = 'https://api.github.com/users/github';

module.exports = async function () {
  try {
    const response = await got.get(URL);
    const data = await response.json();
    console.log({data});
    return data;
  } catch (err) {
    return err;
  }
};

I might try and see if I can shim in in node-fetch@3 if that is supposed to have fixed the issue, but might require a bit of magic to get CommonJS and ESM working together.

[pdehaan]

@b3rimbau Does something like this work? Ref: Dynamic Imports and https://github.com/node-fetch/node-fetch/blob/main/docs/v3-UPGRADE-GUIDE.md#converted-to-es-module

const URL = "https://gateway.euronext.com/api/instrumentDetail";
// const URL = 'https://api.github.com/users/github';

module.exports = async function () {
  const fetch = await import("node-fetch").then(mod => mod.default);
  try {
    const res = await fetch(URL);
    return await res.json();
  } catch (err) {
    process.exitCode = 1;
    return err;
  }
};

It fails for me locally, but that's because I'm not authenticated and the error page seems to return HTML which seems to break the JSON parsing:

SyntaxError: Unexpected token < in JSON at position 0
    at JSON.parse (<anonymous>)
    at Response.json (file:///private/tmp/11ty-2375/node_modules/node-fetch/src/body.js:149:15)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async module.exports (/private/tmp/11ty-2375/src/_data/remote.js:8:12)
    at async TemplateData.getDataValue (/private/tmp/11ty-2375/node_modules/@11ty/eleventy/src/TemplateData.js:494:23)
    at async TemplateData.getAllGlobalData (/private/tmp/11ty-2375/node_modules/@11ty/eleventy/src/TemplateData.js:275:18)
    at async TemplateData.getData (/private/tmp/11ty-2375/node_modules/@11ty/eleventy/src/TemplateData.js:338:24)

[b3rimbau]

@pdehaan Yes, that seems to work - thank you. So this is importing node-fetch@3 whose parser seems better able to cope with the header that is being issued. What is mod => mod.default doing?

Thanks again.
Chris

[pdehaan]

I think without the mod => mod.default trick, I had to use fetch.default(URL) syntax. So I was aliasing it and only returning the default default property to make it more consistent with regular fetch() usage.