Using NextFederationPlugin => Reached heap limit allocation failed

[eroldmrclk]

Describe the bug

Hello,

I would like to explain a problem I encountered during build time.

I can build a large project in a nextjs application with the “next build” command. I added the NextFederationPlugin in next.config.js in this application and run the “next build” command again. After a while I started to get the “reached heap limit allocation failed” error. At the build time, I see that the memory usage in the task manager increases significantly and eventually I get this error. I have tried the issue on several devices and I am sure it is not device related. The only difference between the successful build and the unsuccessful build is the NextFederationPlugin and it is difficult to understand why this occurs.

I did many different tests to find the source of the problem, but I could not reach a clear result. I guess this problem occurs because NextFederationPlugin changes webpack configurations. I can give the following information about the project. There are approximately 100 pages, each page is 1.4mb in size, use SSG (static site generation) and [email protected]. Additionally, when I reduced 100 pages in the project to 50 pages or 100 pages from 1.4mb to 500kb, I was able to get a successful build with NextFederationPlugin.

Is it a problem you've encountered before? Is there anything you can suggest?

Reproduction

Private

Used Package Manager

yarn

System Info

System:
    OS: Windows 10 10.0.19045
    CPU: (8) x64 Intel(R) Core(TM) i7-10610U CPU @ 1.80GHz
    Memory: 9.29 GB / 31.78 GB,
  Binaries:
    Node: 18.14.2 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.19 - C:\Program Files\nodejs\yarn.CMD
    npm: 9.5.0 - C:\Program Files\nodejs\npm.CMD,
  Browsers:
    Edge: Chromium (117.0.2045.43)
    Internet Explorer: 11.0.19041.3636

Validations

• Read the docs.
• Read the common issues list.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Make sure this is a Module federation issue and not a framework-specific issue.
• The provided reproduction is a minimal reproducible example of the bug.

[ScriptedAlchemy]

Increase the memory heap:

NODE_OPTIONS=--max-old-space-size=8192 next dev - give it more memory till it works.

This plugin relies on hacks for next.js in them, one has to do with stats and data which seems to be inefficient on large builds (with ssr enabled)

Workaround for it should be possible if you increase memory.

[eroldmrclk]

Increase the memory heap:

NODE_OPTIONS=--max-old-space-size=8192 next dev - give it more memory till it works.

This plugin relies on hacks for next.js in them, one has to do with stats and data which seems to be inefficient on large builds (with ssr enabled)

Workaround for it should be possible if you increase memory.

Hi Zack,

You are right and thank you for your suggestion. Actually, I knew I would get a successful build if I increased the max limit. In fact, I wonder if you are close to a permanent solution by finding the root cause of this problem and I wanted to give feedback if you are not aware of it. In addition, I think the problem can't be caused by SSR only because I don't use SSR in my large project. (no use getServerSideProps)

Apart from all this, there is one more question I want to ask. Normally, I use the module in the remote application with next/dynamic in a nextjs project, but I wanted to share this usage through an npm package to be used in nextjs applications, but I cannot use next/dynamic in the npm package because I cannot specify that it should use the module in the remote application and I get the module not found error. Instead, I tried to use it with the injectScript I read in your documentation and I was successful. I wonder if there would be a performance difference between next/dynamic and injectScript.

[ScriptedAlchemy]

Inject script doesn't work with v8 use federation/runtime package. Can't use next/dynamic with MFP, use lazy

We have shipped new data plugin but I've not replaced next implementation yet.

[Arkamit]

Hi @ScriptedAlchemy, I'm also getting the same error while trying to build my nextjs project.

Error:

(node:38683) [DEP_WEBPACK_PRE_COMPILED_SCHEMA_INVALID] DeprecationWarning: webpack bug: Pre-compiled schema reports error while real schema is happy. This has performance drawbacks.

<--- Last few GCs --->

[3770:0x591b200] 389049 ms: Scavenge (reduce) 8095.4 (8239.0) -> 8095.2 (8239.7) MB, 14.70 / 0.00 ms (average mu = 0.619, current mu = 0.312) allocation failure;
[3770:0x591b200] 397915 ms: Mark-Compact (reduce) 8096.0 (8239.7) -> 8095.6 (8240.2) MB, 8859.98 / 0.00 ms (average mu = 0.401, current mu = 0.005) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

1: 0xca5580 node::Abort() [node]
2: 0xb781f9 [node]
3: 0xeca4d0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
4: 0xeca7b7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
5: 0x10dc505 [node]
6: 0x10f4388 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
7: 0x10ca4a1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
8: 0x10cb635 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
9: 0x10a8c86 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
10: 0x1503a16 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x7fed8e099ef6

Version of @module-federation/nextjs-mf is 8.2.5

Version of Nextjs is 14.1.0

Projects without module-federation package is building fine.

I have tried increasing the memory like you have mentioned but it is even failing with cross-env NODE_OPTIONS=--max-old-space-size=8192

Also tried increasing the memory to 16GB but it is even failing with cross-env NODE_OPTIONS=--max-old-space-size=16384.

Any suggestions would be appreciated.

@eroldmrclk Were you able to find a solution to this problem? Any help would be appreciated.

[github-actions]

Stale issue message

[ScriptedAlchemy]

--max-old-space-size with no number will allow node to consume all memory needed.

[VisalakshiVairavan]

@ScriptedAlchemy, my app takes more than 30 Gigs memory to build with the plugin. Is it possible to suggest another workaround as using that much memory is not practical.

[zackarychapple]

@VisalakshiVairavan how big is the app that you need 30 Gigs of memory to build? Perhaps you will need to decompose it into smaller modules?

[ScriptedAlchemy]

Plugin needs to be rewritten. Even then no promise anything improves for next.

[Gu7z]

Are there any plans to proceed with the rewrite of the plugin that was mentioned? If so, is there a timeline for when this might be implemented?

[ScriptedAlchemy]

Not currently.
PR can be sent, remove ChunkCorrelationPlugin, use the built in mf-manifest for chunk correlation information, then reimplement the chunk flushing functions to use the mf-manifest instead of the existing plugin.

[ScriptedAlchemy]

Warning Support for next is ending

#3153