Build Fails with Large Number of Markdown Files (100k+) in Astro Project

[pantickatorce]

Astro Info

✅ Astro 5.1.0
✅ Node v20.17.0

If this issue only occurs in one browser, which browser is a problem?

No response

Describe the Bug

I'm struggling with an issue when running npm run build locally. The process fails, and I'm at my wit's end trying to fix it. My project has more than 100,000 .md files in the content folder, and I use getCollection("posts") in several places. I've already tried increasing memory limits with --max-old-space-size=12288 and many other tweaks, but nothing seems to work.

The website is intended to be fully static on the server, and I need the entire dist folder to be generated.

Is there any solution for this? I’m really desperate at this point.

Thank you so much in advance!

Console:

`23:47:12 [build] Collecting build info...
23:47:12 [build] ✓ Completed in 37.32s.
23:47:12 [build] Building static entrypoints...

<--- Last few GCs --->

[18644:000002071B902880]   216506 ms: Mark-Compact (reduce) 4080.4 (4138.7) -> 4080.4 (4139.2) MB, 654.26 / 0.02 ms  (+ 98.4 ms in 10 steps since start of marking, biggest step 1
= 0.545, current [18644:000002071B902880]   218604 ms: Mark-Compact (reduce) 4096.9 (4155.2) -> 4086.6 (4145.4) MB, 1847.77 / 0.00 ms  (average mu = 0.305, current mu = 0.119) al


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 00007FF762A239EB node::SetCppgcReference+18459
 2: 00007FF762993238 DSA_meth_get_flags+93160
 3: 00007FF76340C771 v8::Isolate::ReportExternalAllocationLimitReached+65
 4: 00007FF7633F5EC8 v8::Function::Experimental_IsNopFunction+1336
 5: 00007FF763257970 v8::Platform::SystemClockTimeMillis+659328
 6: 00007FF7632549F8 v8::Platform::SystemClockTimeMillis+647176
 7: 00007FF763269D0A v8::Platform::SystemClockTimeMillis+733978
 8: 00007FF76326A587 v8::Platform::SystemClockTimeMillis+736151
criptor::writable+678094
12: 00007FF6E3C2735D
error: script "build" exited with code 134`

What's the expected result?

All dist archives

Link to Minimal Reproducible Example

https://pantic14.com/

Participation

• I am willing to submit a pull request for this issue.

[Yuelioi]

I have the same issue, so I decided to use dynamic rendering, just like WordPress.

[ematipico]

@pantickatorce have you considered using the new content collections using the new loader API? https://docs.astro.build/en/guides/content-collections/#built-in-loaders

[ascorbic]

If the site is using 5.1.0 then it will be using content layer unless specifically disabled, so I don't think there's much scope for improvement there.

We need a reproduction if we're to do anything about this, though I think that it may end up just being that 100k pages needs lots of memory whatever's happening.

[github-actions]

Hello @pantickatorce. Please provide a minimal reproduction using a GitHub repository or StackBlitz. Issues marked with needs repro will be closed if they have no activity within 3 days.