Getting "violation of fair-usage policy" when starting the app with an empty cache

[bamarch]

When starting the server it will download scripts from the CDN

When starting with an empty directory, the number of generated requests sent to the CDN triggers "violation of fair-usage policy" errors

It feels like the server should be able to download all the scripts it needs in one go

Perhaps there is a way for us to ask the server to download these scripts more slowly, do some rate-limiting?

[WillForrestMindBridge]

Hi I ran into this issue today as well.

I've tried to trace the root cause of the issue, it appears that the library being used to make download requests is using default settings, and this may not be preferred by the Fair use policy

The export server is currently using a nodeJS http proxy to download files.

It:

• Loads files one at a time (No batching)
• May or may not have a user agent that's allowed by the fair use policy
• May or may not have the http-referrer
• Fails to cache the files unless you get all files described in the manifest for a highcharts version
  • So if it half-fails, you have to restart from scratch, causing even more download requests, on the next run of the server

The proxy agent is built here with no options.

The cache check is here where it will re-force downloads if any module is wrong.

Was there a change to make the fair use policy a bit more strict in regards to the user agent?

[pjanaya]

I also ran into this issue yesterday. It's really annoying and, honestly, worrisome. We bundle the export server with the version of Highcharts we want to use, so there is absolutely no need to go fetch anything. I couldn't find any option to let the export server know "please, use the Highcharts files from node_modules instead of fetching them". This blocked our CI/CD system for a whole day, since we couldn't even pass out tests. The fact that Highcharts can, unilaterally, break our systems is mindblowing.

This is what I ended up doing. I'm not proud of it, but it's the only solution I could come up with. I created a Local CDN Server just for Highcharts. Here's the code in case someone can benefit from it.

import express, { Response } from 'express';
import * as http from 'http';
import { readFileSync } from 'fs';
import path from 'path';

let server: http.Server;

interface LocalHighchartsCDNServerInterface {
  start: () => void;
  close: () => void;
}

const LocalHighchartsCDNServer: LocalHighchartsCDNServerInterface = {
  start: () => {
    const app = express();

    const sendFile = (
      req: { params: { version: string; filename: string } },
      res: Response,
      basePath: string,
    ) => {
      const filePath = path.join(path.resolve(), basePath, req.params.filename);

      res.status(200).send(readFileSync(filePath));
    };

    app.get('/cdn/:version/:filename', (req, res) => {
      sendFile(req, res, 'node_modules/highcharts/');
    });

    app.get('/cdn/:version/modules/:filename', (req, res) => {
      sendFile(req, res, 'node_modules/highcharts/modules/');
    });

    app.get('/cdn/maps/:version/modules/:filename', (req, res) => {
      sendFile(req, res, 'node_modules/highcharts/modules/');
    });

    server = app.listen(8080, () => {
      console.log(`[server]: Server is running at http://localhost:${8080}`);
    });
  },
  close: () => {
    if (server) {
      server.close();
    }
  },
};

export default LocalHighchartsCDNServer;

Then, I specify this URL as the cdnURL:

highcharts: {
    version: '11.4.8', // When using the local CDN, this version string is not used. It uses whatever you have in node_modules
    forceFetch: false,
    cdnURL: 'http://localhost:8080/cdn/',
    customScripts: [],
    indicatorScripts: [],
},

Notice how I'm passing an empty array to customScripts and indicatorScripts, since I'm not using them. This might not be the case for everyone.

Then, we call LocalHighchartsCDNServer.start(); before the export. In our case, before doing exporter.initExport(exportSettings).

[lewis-jackson-bots]

We're seeing the exact same issue without any change in our dependencies or the number of modules we need to download (~60).

Fails to cache the files unless you get all files described in the manifest for a highcharts version

So if it half-fails, you have to restart from scratch, causing even more download requests, on the next run of the server

We noticed this as well, it means that we can't attempt to incrementally build the cache by starting up the server multiple times.

I've tried to trace the root cause of the issue, it appears that the library being used to make download requests is using default settings, and this may not be preferred by the Fair use policy

I think that this policy may refer to another product, but it could well be the same as the https://code.highcharts.com/ policy? It seems to refer to a hosted version of the export server. If the headers are the issues, we could potentially create a proxy which adds them.

Why does the CDN not advertise the rate limit in its response headers? We don't know if we're allowed to make zero requests, one request, or 30 requests. We don't know the window of the rate limit. We only get told the retry-after once we've broken the rate limit.

Was there a change to make the fair use policy a bit more strict in regards to the user agent?

I think that this must be the case. We don't build our project using HighCharts often, but we had a successful run on the 10th of February.

Workarounds we've considered:

• uploading the cached sources.js file to the repo, grab this from an older build
• run our own CDN out of cloud storage
• as mentioned above, a proxy which adds the correct headers to CDN requests
• there's a couple of other options in this thread:
  • Fetch Highcharts from CDN during docker build #415 (comment)
  • Fetch Highcharts from CDN during docker build #415 (comment)

[pjanaya]

@lewis-jackson-bots We really shouldn't need any workarounds for those of us who want to use the Highcharts version we have in node_modules. Unless I'm missing something, I couldn't find any combination of configurations or settings that would stop the export server from fetching all files from the CDN and use the local version instead. Because even if we add the headers to the request, we are still relying on the Highcharts

As mentioned in their Fair use policy:

Please note that due to the variability of traffic, there are no fixed limits on our service. Limits fluctuate in real time based on current traffic, complexity, and system load.

So, at any point, they might detect a high load and block everyone. Interfering with people's builds is unacceptable when there is no need to fetch anything.

[lewis-jackson-bots]

We really shouldn't need any workarounds for those of us who want to use the Highcharts version we have in node_modules.

Agreed, but I have no power to control that. All we can do is complain and add a workaround in the meantime.

[melsdezeeuw]

We are facing the same issue since today an I already have created a support ticket for this as well

[daddyg]

We have the same issues. My understanding is that the fair use policy is supposed to protect the officially hosted export server from heavy usage on the export.highcharts.com domain.

It looks like this fair usage policy has recently been applied to the code.highcharts.com domain as well. Since this is supposed to be a CDN I can only assume this is a mistake on HighCharts part and will be fixed in time 🙏

In the meantime, the easiest fix I have found is by setting the HIGHCHARTS_CDN_URL environment variable to the cloudflare mirror

set HIGHCHARTS_CDN_URL=https://cdnjs.cloudflare.com/ajax/libs/highcharts/

[WillForrestMindBridge]

In the meantime, the easiest fix I have found is by setting the HIGHCHARTS_CDN_URL environment variable to the cloudflare mirror

This is a rather nice workaround, and stops us from hosting an additional express server.

[PaulDalek]

Hi all,

Thank you for reporting this, and apologies for the inconvenience. This issue is due to recent changes in the fair usage policy and rate limiting of incoming requests to https://code.highcharts.com/.

I agree that the current implementation of script fetching and caching is not ideal, and there is room for improvement. This mainly includes adding an option to use the scripts stored locally (e.g., from the npm package). The 'fetch per module' approach was implemented this way due to the possibility of combining modules, indicators, and custom scripts that users can select.

Additionally, while the public export server requires the Referer and User-Agent headers when requesting an export, the script-fetching request does not (at least for now).

Although it has worked fine up until now, I recommend not relying on the public Highcharts CDN in a production environment. For now, some possible workarounds include using free CDNs like jsDelivr or CDNJS (I know it’s not perfect, but it would allow the initial script fetching for the server and is not as rate-limited, so it should be fine for now), or using previously generated cache content. Additionally, thank you for all your suggestions and workarounds!

I'll mark this as an enhancement to implement npm script usage.

[lewis-jackson-bots]

We have the same issues. My understanding is that the fair use policy is supposed to protect the officially hosted export server from heavy usage on the export.highcharts.com domain.

It looks like this fair usage policy has recently been applied to the code.highcharts.com domain as well. Since this is supposed to be a CDN I can only assume this is a mistake on HighCharts part and will be fixed in time 🙏

In the meantime, the easiest fix I have found is by setting the HIGHCHARTS_CDN_URL environment variable to the cloudflare mirror

set HIGHCHARTS_CDN_URL=https://cdnjs.cloudflare.com/ajax/libs/highcharts/

This is a great suggestion, unfortunately it doesn't support two files that are grabbed by default (unsure if we need them):

• map.js
• indicators-all.js

[lewis-jackson-bots]

Thank you for reporting this, and apologies for the inconvenience. This issue is due to recent changes in the fair usage policy and rate limiting of incoming requests to https://code.highcharts.com/.

Can the rate limit be tweaked to allow us to at least start the app with the default modules once per 15 minutes for example? At the moment the code in this repo can't be run by anyone.

[bamarch]

This is a great suggestion, unfortunately it doesn't support two files that are grabbed by default (unsure if we need them):

map.js
indicators-all.js

Hit that too...

Happy now with a riff on pjanaya solution:
#633 (comment)

// start_express_cdn.js
// Runs a simple Express server that serves Highcharts JS files from node_modules on local fs

const express = require('express');
const path = require('path');
const fs = require('fs');

const app = express();
const port = 9080;

const sendFile = (req, res, basePath) => {
  const fileLocation = req.params.dirname ? `${req.params.dirname}/${req.params.filename}` : req.params.filename;
  const filePath = path.join(process.cwd(), basePath, fileLocation);
  
  console.log(`[server]: ${req.url} -> ${basePath}${req.params.filename}`);
  try {
    const fileContent = fs.readFileSync(filePath);
    res.status(200).send(fileContent);
  } catch (error) {
    res.status(404).send(`File not found: ${req.params.filename}`);
  }
};

// e.g. http://localhost:9080/cdn/12.1.2/highcharts.js
// -> ./node_modules/highcharts/highcharts.js
app.get('/cdn/:version/:filename', (req, res) => {
  sendFile(req, res, 'node_modules/highcharts/');
});

// modules
// e.g. http://localhost:9080/cdn/12.1.2/modules/exporting.js
// -> ./node_modules/highcharts/modules/exporting.js
app.get('/cdn/:version/modules/:filename', (req, res) => {
  sendFile(req, res, 'node_modules/highcharts/modules/');
});

// maps modules
// e.g. http://localhost:9080/cdn/maps/12.1.2/modules/map.js
// -> ./node_modules/highcharts/modules/map.js
app.get('/cdn/maps/:version/modules/:filename', (req, res) => {
  sendFile(req, res, 'node_modules/highcharts/modules/');
});

// stock
// e.g. http://localhost:9080/cdn/stock/12.1.2/indicators/indicators-all.js
// -> ./node_modules/highcharts/indicators/indicators-all.js
app.get('/cdn/stock/:version/:dirname/:filename', (req, res) => {
  sendFile(req, res, 'node_modules/highcharts/');
});

// Only start the server if this file is run directly
if (require.main === module) {
  const server = app.listen(port, () => {
    console.log(`[server]: Express CDN server is running at http://localhost:${port}`);
  });

  // Handle graceful shutdown
  process.on('SIGTERM', () => {
    console.log('SIGTERM signal received: closing HTTP server');
    server.close(() => {
      console.log('HTTP server closed');
      process.exit(0);
    });
  });
}

// Still allow importing as a module if needed
module.exports = app;

#!/bin/sh
# run_and_stop_server_to_populate_cache.sh
# POSIX (support different containers and shells)

# this is used in docker build to ensure scripts from CDN are already cached (otherwise our firewall will block the downloads in prod)
# https://github.com/highcharts/node-export-server/issues/415#issuecomment-2041409265

INSTALL_DIR=$1

(cd ./cdn && npm install highcharts@"${HIGHCHARTS_VERSION}" --prefix .)
(cd ./cdn && npm install [email protected] --prefix .)
(cd ./cdn && node start_express_cdn.js &)
pid_express_cdn=$!

highcharts-export-server --enableServer "1" &
pid_export_server=$!
until test -f "$INSTALL_DIR"/.cache/manifest.json && test -f "$INSTALL_DIR"/.cache/sources.js; do
  sleep 1
done

kill $pid_express_cdn
kill $pid_export_server

...
ENV HIGHCHARTS_VERSION=12.1.2
ENV HIGHCHARTS_CDN_URL=http://localhost:9080/cdn/
RUN ./scripts/run_and_stop_server_to_populate_cache.sh ${INSTALL_DIR}
...

Cutting out the highcharts CDNs to use something local, populated from NPM at build-time

May have to maintain the list of mappings when upgrading versions, but don't mind the trade-off

[bamarch]

I'll mark this as an enhancement to implement npm script usage.

It does seem a bit more urgent than an enhancement, since the app now cannot run using its default configuration

I would suggest it is more like a bug

some possible workarounds include using free CDNs like jsDelivr or CDNJS

Perhaps this should become the default for HIGHCHARTS_CDN_URL?

[PaulDalek]

Hi,

Regarding this from my message yesterday:

Additionally, while the public export server requires the Referer and User-Agent headers when requesting an export, the script-fetching request does not (at least for now).

Now it does. A hotfix has been applied to the master branch (likely to be published on npm tomorrow) that lifts the restriction on script fetching. This might be a temporary solution until the implementation of local script usage.