yarn-managed monorepo, building from source, nodeLinker=node-modules: The nearest package directory doesn't seem to be part of the project

[XilinJia]

in my monorepo project:

✗ yarn install
YN0007: │ sharp@npm:0.33.2 must be built because it never has been before or the last one failed
YN0009: │ sharp@npm:0.33.2 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-4494d0f3/build.log)

/tmp/xfs-4494d0f3/build.log:

This file contains the result of Yarn building a package (sharp@npm:0.33.2)

Script name: install

sharp: Detected globally-installed libvips v8.15.1
sharp: Attempting to build from source via node-gyp
sharp: Found node-addon-api
sharp: Found node-gyp version 10.0.1
sharp: See https://sharp.pixelplumbing.com/install#building-from-source
�[31m�[1mUsage Error�[22m�[39m: The nearest package directory (/myProject/packages/lib/node_modules/sharp) doesn't seem to be part of the project declared in /myProject.

• If /myProject isn't intended to be a project, remove any yarn.lock and/or package.json file there.
• If /myProject is intended to be a project, it might be that you forgot to list packages/lib/node_modules/sharp in its workspace configuration.
• Finally, if /myProject is fine and you intend packages/lib/node_modules/sharp to be treated as a completely separate project (not even a workspace), create an empty yarn.lock file in it.

�[1m$ �[22myarn run [--inspect] [--inspect-brk] [-T,--top-level] [-B,--binaries-only] [--require #0] ...

[lovell]

The error message from yarn is trying to help you - watch out for any extra or stray lockfiles - see yarnpkg/berry#2212

[XilinJia]

Well, I thought the yarn message would help you.

If you look at the message in detail, it tells me to put a yarn.lock file in node_modules/sharp, which is not what I should do.

BTW, using version "sharp": "0.32.6" doesn't have this problem.

[lovell]

Please can you provide a complete, minimal repo with all the directories, package.json and lockfiles that allows someone else to reproduce.

[XilinJia]

I'm not sure how to make a minimal repo, but the issue occurs in my monorepo project Xilinota https://github.com/XilinJia/Xilinota. If someone wants to look into it, the root dir has yarn.lock and none of the packages have. A simple package for someone to check that depends on sharp is packages/tools (currently set to use 0.32.6 though).

[lovell]

sharp: Detected globally-installed libvips v8.15.1
sharp: Attempting to build from source via node-gyp

You're building sharp from source against a globally-installed libvips. There are 3 packages within the monorepo that have a dependency on sharp. Running yarn attempts to build these in parallel.

When I run this locally I see 2 of these fail and 1 succeed, which points to a race condition within yarn and its node_modules linker.

If I change the hoisting limits configuration:

-nmHoistingLimits: workspaces
+nmHoistingLimits: dependencies

..then it installs correctly.

[XilinJia]

Thanks for the analysis.

In what ways can I keep the workspaces setting? And what's different with 0.32.6?

[lovell]

My understanding of yarn is that it detects native modules (presence of binding.gyp in the package root, a dependency on node-gyp etc.) and controls build concurrency accordingly.

From sharp v0.33.0 onward the ability to build from source is opt-it and you need to signal this to yarn - see #3750

As you're building sharp from source, you'll need to add node-addon-api and node-gyp as dependencies alongside it in any sub-packages.

https://sharp.pixelplumbing.com/install#building-from-source

[lovell]

@XilinJia Were you able to make progress with this?

[lovell]

Closing due to inactivity but please feel free to reopen with more details if further help is required.

[tyteen4a03]

Can confirm this still happens in Yarn 4.1.1, and that the nmHoistingLimits: dependencies fixes the issue, but also causes issues with Next.js not being able to resolve their packages. Interestingly, after changing it back to nmHoistingLimits: workspaces and running yarn, there were no further complaints from sharp.

If it helps, I am using PayloadCMS which might simulate a real-world monorepo setup better.