Bazel Memory Tracker shows incorrect memory consumption #24782

AlexanderGolovlev · 2024-12-20T14:41:53Z

Description of the bug:

According to documentation Bazel allows to instrument memory allocations. It seems that now this feature provides an incorrect information.
All values of memory consumption for rules are either 0 or values which are roughly multiples of 256kB. For example, the text dump for prof.gz shows:

Showing nodes accounting for 5888.52kB, 100% of 5888.52kB total
      flat  flat%   sum%        cum   cum%
 4352.59kB 73.92% 73.92%  4352.59kB 73.92%  repository_rule <builtin>
  512.02kB  8.70% 82.61%   512.02kB  8.70%  config_setting <builtin>
  256.07kB  4.35% 86.96%   256.07kB  4.35%  create_linking_context_from_compilation_outputs <builtin>
  255.99kB  4.35% 91.31%   255.99kB  4.35%  cc_toolchain_features <builtin>
  255.93kB  4.35% 95.65%   255.93kB  4.35%  cc_object_library <builtin>
...

The total memory consumption is also far from expected values.
It looks like the values of memory consumption are taken from shifted position in memory. It might be caused by changes in internal Java structures.

I believe, the problem is in outdated version of java-allocation-instrumenter-3.3.0.jar used for instrumenting. According to Release notes the support for Java 17 was added in 3.3.2, and support for Java 21 was added in 3.3.4.
However, my attempts to use a newer version of instrumenter were unsuccessful. Bazel crashes with any version higher than 3.3.0. The error message is unclear:

Exception in thread "main" java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(Unknown Source)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown Source)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException: Cannot invoke "java.net.URL.openConnection()" because "url" is null
	at com.google.monitoring.runtime.instrumentation.AllocationInstrumenterBootstrap.premain(AllocationInstrumenterBootstrap.java:50)
	... 4 more
Caused by: java.lang.NullPointerException: Cannot invoke "java.net.URL.openConnection()" because "url" is null
	at com.google.monitoring.runtime.instrumentation.AllocationInstrumenterBootstrap.premain(AllocationInstrumenterBootstrap.java:40)
	... 4 more
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message Outstanding error when calling method in invokeJavaAgentMainMethod at s\src\java.instrument\share\native\libinstrument\JPLISAgent.c line: 627
*** java.lang.instrument ASSERTION FAILED ***: "success" with message invokeJavaAgentMainMethod failed at s\src\java.instrument\share\native\libinstrument\JPLISAgent.c line: 466
*** java.lang.instrument ASSERTION FAILED ***: "result" with message agent load/premain call failed at s\src\java.instrument\share\native\libinstrument\JPLISAgent.c line: 429
FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed

Which category does this issue belong to?

Performance

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Perform steps according to documentation:

Pass STARTUP_FLAGS to any Bazel invocation:

  STARTUP_FLAGS=\
  --host_jvm_args=-javaagent:<path to java-allocation-instrumenter-3.3.0.jar> \
  --host_jvm_args=-DRULE_MEMORY_TRACKER=1

Run any bazel build
Run bazel dump --rules or bazel dump --skylark_memory=$HOME/prof.gz

Which operating system are you running Bazel on?

Windows, Linux, Mac

What is the output of `bazel info release`?

release 7.4.0

If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

No response

What's the output of `git remote get-url origin; git rev-parse HEAD` ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

The text was updated successfully, but these errors were encountered:

fmeum · 2024-12-20T19:34:12Z

AllocationTrackerModule forces a sample size of 256 KB, which results in all measurements being a multiple of that. I updated the version in #24783, but I can't reproduce the crash with it (on macOS). Could you try again with Bazel from that PR?

AlexanderGolovlev · 2024-12-23T13:24:14Z

Thank you, @fmeum
I tried with your changes applied over Bazel 7.4.0 and java-allocation-instrumenter-3.3.4.jar. It doesn't crash any more. Also, memory consumption now looks much more real than before. For the same target as in initial comment, it shows 106.48Mb instead of 5.88Mb:

Showing nodes accounting for 94.42MB, 88.72% of 106.42MB total
Dropped 362 nodes (cum <= 0.53MB)
      flat  flat%   sum%        cum   cum%
   58.94MB 55.38% 55.38%    58.94MB 55.38%  create_compilation_context <builtin>
      15MB 14.10% 69.48%       15MB 14.10%  repository_rule <builtin>
    3.76MB  3.54% 73.01%     3.76MB  3.54%  create_linking_context_from_compilation_outputs <builtin>
    3.71MB  3.48% 76.50%     3.71MB  3.48%  depset <builtin>
    3.50MB  3.29% 79.79%     3.50MB  3.29%  glob <builtin>
    1.50MB  1.41% 81.20%     1.50MB  1.41%  format <builtin>
    1.25MB  1.17% 82.37%     1.25MB  1.17%  register_toolchains <builtin>
    1.25MB  1.17% 83.55%     1.25MB  1.17%  declare_file <builtin>
    1.25MB  1.17% 84.72%     1.25MB  1.17%  filegroup <builtin>
    1.01MB  0.94% 85.67%     2.51MB  2.36%  compile <builtin>
       1MB  0.94% 86.61%        1MB  0.94%  precompiled_headers <builtin>
...

It seems that this PR fully resolves the issue.

The jar can be supplied separately as it needs to be passed to the `-javaagent` JVM arg anyway. Also update the recommended version for compatibility with Java 21. Fixes bazelbuild#24782 Closes bazelbuild#24783. PiperOrigin-RevId: 714057306 Change-Id: Ic8d47562f15344ed16d0af5d3242937d2ee53b22 (cherry picked from commit b59fbee)

…#24912) The jar can be supplied separately as it needs to be passed to the `-javaagent` JVM arg anyway. Also update the recommended version for compatibility with Java 21. Fixes #24782 Closes #24783. PiperOrigin-RevId: 714057306 Change-Id: Ic8d47562f15344ed16d0af5d3242937d2ee53b22 (cherry picked from commit b59fbee) Fixes #24788

The jar can be supplied separately as it needs to be passed to the `-javaagent` JVM arg anyway. Also update the recommended version for compatibility with Java 21. Fixes bazelbuild#24782 Closes bazelbuild#24783. PiperOrigin-RevId: 714057306 Change-Id: Ic8d47562f15344ed16d0af5d3242937d2ee53b22

The jar can be supplied separately as it needs to be passed to the `-javaagent` JVM arg anyway. Also update the recommended version for compatibility with Java 21. Fixes bazelbuild#24782 Closes bazelbuild#24783. PiperOrigin-RevId: 714057306 Change-Id: Ic8d47562f15344ed16d0af5d3242937d2ee53b22 (cherry picked from commit b59fbee)

…#24913) The jar can be supplied separately as it needs to be passed to the `-javaagent` JVM arg anyway. Also update the recommended version for compatibility with Java 21. Fixes #24782 Closes #24783. PiperOrigin-RevId: 714057306 Change-Id: Ic8d47562f15344ed16d0af5d3242937d2ee53b22 (cherry picked from commit b59fbee) Fixes #24789

iancha1992 · 2025-01-22T23:58:45Z

A fix for this issue has been included in Bazel 7.5.0 RC2. Please test out the release candidate and report any issues as soon as possible.
If you're using Bazelisk, you can point to the latest RC by setting USE_BAZEL_VERSION=7.5.0rc2. Thanks!

AlexanderGolovlev added type: bug untriaged labels Dec 20, 2024

AlexanderGolovlev assigned iancha1992, satyanandak and sgowroji Dec 20, 2024

github-actions bot added the team-Performance Issues for Performance teams label Dec 20, 2024

sgowroji unassigned sgowroji, iancha1992 and satyanandak Dec 20, 2024

fmeum mentioned this issue Dec 20, 2024

Remove allocation instrumenter jar and update version in docs #24783

Closed

oquenchil added P2 We'll consider working on this in future. (Assignee optional) and removed untriaged labels Jan 7, 2025

copybara-service bot closed this as completed in b59fbee Jan 10, 2025

fmeum mentioned this issue Jan 13, 2025

[7.5.0] Remove allocation instrumenter jar and update version in docs #24912

Merged

fmeum mentioned this issue Jan 13, 2025

[8.1.0] Remove allocation instrumenter jar and update version in docs #24913

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bazel Memory Tracker shows incorrect memory consumption #24782

Bazel Memory Tracker shows incorrect memory consumption #24782

AlexanderGolovlev commented Dec 20, 2024

fmeum commented Dec 20, 2024

AlexanderGolovlev commented Dec 23, 2024 •

edited

Loading

iancha1992 commented Jan 22, 2025

Bazel Memory Tracker shows incorrect memory consumption #24782

Bazel Memory Tracker shows incorrect memory consumption #24782

Comments

AlexanderGolovlev commented Dec 20, 2024

Description of the bug:

Which category does this issue belong to?

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Which operating system are you running Bazel on?

What is the output of bazel info release?

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

What's the output of git remote get-url origin; git rev-parse HEAD ?

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

Have you found anything relevant by searching the web?

Any other information, logs, or outputs that you want to share?

fmeum commented Dec 20, 2024

AlexanderGolovlev commented Dec 23, 2024 • edited Loading

iancha1992 commented Jan 22, 2025

What is the output of `bazel info release`?

If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

What's the output of `git remote get-url origin; git rev-parse HEAD` ?

AlexanderGolovlev commented Dec 23, 2024 •

edited

Loading