Add documentation on how to test JavaFX app

[dlemmermann]

Can you provide an example on how to setup a test for a JavaFX app / control? Especially one that also works with the module system.

[FlorianKirmaier]

I've added a lot of real world test samples to the Readme.

[danielpeintner]

Hi Florian,

Thank you very much for your work in getting rid of memory leaks!

I did see your links about real word examples (FYI some links are broken like the one for SpaceFX) but what I still fail to do is a very simple JavaFX test.

Let's assume one creates a JavaFX Pane and after a while the pane gets invisible (i.e. not used anymore). I wanted to create a very simple test to check whether it is possible to detect whether the node (and maybe also its FXML controller) can be properly garbage collected.

Attached a very simple SSCCE example (without FXML) that tries to check based on testfx and Junit5 whether a pane is collectable after it gets removed. It always reports an error.
Can you give me a hint what I am doing wrong. Thanks!

import de.sandec.jmemorybuddy.JMemoryBuddy;
import javafx.application.Platform;
import javafx.scene.Node;
import javafx.scene.Scene;
import javafx.scene.control.Button;
import javafx.scene.layout.BorderPane;
import javafx.scene.layout.Pane;
import javafx.stage.Stage;
import org.junit.jupiter.api.Test;
import org.testfx.framework.junit5.ApplicationTest;

import static org.junit.jupiter.api.Assertions.assertNotNull;

public class MemoryPaneTest extends ApplicationTest {

    Scene scene;
    BorderPane bp;
    Node centerNode;

    @Override
    public void start(Stage stage) throws Exception {
        this.bp = new BorderPane();
        bp.setCenter(this.centerNode = getCenterNode());
        scene = new Scene(bp, 800, 600);
        stage.setScene(scene);
        stage.show();
    }

    protected Node getCenterNode() throws Exception {
        BorderPane p = new BorderPane();
        bp.setCenter(new Button("ABC"));
        return p;
    }

    @Test
    public void check1() {
        assertNotNull(scene);
        assertNotNull(centerNode);

        JMemoryBuddy.memoryTest(checker -> {
            checker.assertNotCollectable(centerNode); // should not be collectable, still visible
        });

        // remove pane from scene
        Platform.runLater(() -> {
            this.bp.getChildren().remove(this.centerNode);
            this.bp.setCenter(new Pane());

            // checking here fails with --- Exception in Async Thread ---
        });

        try {
            Thread.sleep(1000);
        } catch (InterruptedException e) {
        }

        JMemoryBuddy.memoryTest(checker -> {
            checker.assertCollectable(centerNode); // fails always
        });
    }
}

[FlorianKirmaier]

The test doesn't work, because as long as the Test is running, all it's classmembers cannot be collected, so checking whether they are collectable always fails.

I wouldn't use class members, when writing a memory test. It's better to put all variables in the Lambda of the memoryTest, this way they don't interfere with the test. If you do so anyways, then you would have to set them to null.
Some test frameworks have the tendency to be not really "memorytest-friendly" - so I usually avoid TestFX. Some mocking frameworks might also have issues - in some unknown situations.

Maybe this test is a better reference? https://github.com/openjdk/jfx/blob/fdc88341f1df8fb9c99356ada54b25124b77ea6e/tests/system/src/test/java/test/javafx/scene/control/ProgressIndicatorLeakTest.java

[danielpeintner]

Maybe this test is a better reference? https://github.com/openjdk/jfx/blob/fdc88341f1df8fb9c99356ada54b25124b77ea6e/tests/system/src/test/java/test/javafx/scene/control/ProgressIndicatorLeakTest.java

Yes, this really helped a lot 👍

I followed that approach (i.e., the spirit of stageLeakWhenTreeNotShowing()).

The surprising part to me was that I easily created a memory leak when adding a MenuButton. Without MenuButton JMemoryBuddy can collect the stage while when adding an empty MenuButton I get an error. Am I using it wrongly or is it really that easy to create leaks with JavaFX ?
see example below which uses Junit5 but you can essentially copy it to ProgressIndicatorLeakTest.java you provided and update the assert calls.

    @Test
    public void stageLeakWithMenuButton() throws Exception {
        JMemoryBuddy.memoryTest((checker) -> {
            CountDownLatch showingLatch = new CountDownLatch(1);
            AtomicReference<Stage> stage = new AtomicReference<>();

            TestUtil.runAndWait(() -> {
                stage.set(new Stage());
                BorderPane root = new BorderPane();

                ToolBar tb = new ToolBar();
                root.setTop(tb);
                tb.getItems().add(new Button("A"));
                tb.getItems().add(new Button("B"));
                tb.getItems().add(new Button("C"));
                // causes memory leak !!!
                tb.getItems().add(new MenuButton("MB"));

                stage.get().setScene(new Scene(root));
                stage.get().setOnShown(l -> {
                    Platform.runLater(() -> showingLatch.countDown());
                });
                stage.get().show();
            });

            try {
                assertTrue(showingLatch.await(15, TimeUnit.SECONDS), "Timeout waiting test stage");
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }

            TestUtil.runAndWait(() -> {
                stage.get().close();
            });

            checker.assertCollectable(stage.get());
        });
    }

[FlorianKirmaier]

Before investigating it, which version of JavaFX are you using?
You should use at least JavaFX16, otherwise, you might run into leaks that are already fixed.
It's also important to not use monocle - because of a leak fixed some time ago - and then reverted due to issues with android.

It's worth mentioning, that many of these leaks are only noticed when writing such precise tests. In production, many leaks don't cause big issues because they don't build up over time.

[FlorianKirmaier]

If i would guess - populating the native menu-bar might have a leak and remembers the last stage it was used with. Not a big problem in production - but with this kind of tests such bugs are easily found, fixed, and secured against regression.

[danielpeintner]

Before investigating it, which version of JavaFX are you using?
You should use at least JavaFX16, otherwise, you might run into leaks that are already fixed.

FYI: I am using 17-ea+16

If i would guess - populating the native menu-bar might have a leak and remembers the last stage it was used with. Not a big problem in production - but with this kind of tests such bugs are easily found, fixed, and secured against regression.

I agree. The only problem is with such minor memory leaks it is more difficult to find the more crucial ones..

[FlorianKirmaier]

This test works for me:

    @Test
    public void stageLeakWithMenuButton() throws Exception {
        JMemoryBuddy.memoryTest((checker) -> {
            CountDownLatch startupLatch = new CountDownLatch(1);
            CountDownLatch showingLatch = new CountDownLatch(1);
            AtomicReference<Stage> stage = new AtomicReference<>();

            Platform.startup(() -> {
                startupLatch.countDown();
            });
            try {
                startupLatch.await(15, TimeUnit.SECONDS);
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }

            Platform.runLater(() -> {
                stage.set(new Stage());
                BorderPane root = new BorderPane();

                ToolBar tb = new ToolBar();
                root.setTop(tb);
                tb.getItems().add(new Button("A"));
                tb.getItems().add(new Button("B"));
                tb.getItems().add(new Button("C"));
                // causes memory leak !!!
                tb.getItems().add(new MenuButton("MB"));

                stage.get().setScene(new Scene(root));
                stage.get().setOnShown(l -> {
                    Platform.runLater(() -> showingLatch.countDown());
                });
                stage.get().show();
            });

            try {
                showingLatch.await(15, TimeUnit.SECONDS);
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }

            Platform.runLater(() -> {
                stage.get().close();
            });

            checker.assertCollectable(stage.get());
        });
    }

So I guess MenuBar is fine.
Ironically I found another leak.
When calling Platform.startup, the provided lambda is never collected. I guess I find them faster as I can fix them. : /

[danielpeintner]

Mhh, interesting. In my case the test you provided also fails.

java.lang.AssertionError: The following references should be collected: [javafx.stage.Stage@7674b62c]

Question: Do you have any other boiler-code in the file since you removed initFX() right? Can you share the full file?
OR are you saying the error comes from the lambda that is not collected...

I am using 'de.sandec:JMemoryBuddy:0.5.1' which seeems the latest version also.

[danielpeintner]

FYI: I put the test how I use it on GitHub for your convenience.

Since GH is "Unable to open DISPLAY" it fails. Locally it fails with the afore mentioned message.

[FlorianKirmaier]

I've added the whole test. Test should be green. But these tests might also be platform-dependent. I've run it on Mac.
QuickTest.java.txt

Yes, being Unable to open DISPLAY is a common error on CI with JavaFX.
The following snippet creates a "virtual display":
export DISPLAY=:99.0 && /sbin/start-stop-daemon --start --quiet --pidfile /tmp/custom_xvfb_99.pid --make-pidfile --background --exec /usr/bin/Xvfb -- :99 -ac -screen 0 1280x1024x16;

The test you've provided works on my machine. So the next step would be to look into the Heapdump and check the GC path.

[danielpeintner]

I see. Having said hat, this seems to be somewhat operating system dependent which is weird.
Is the platform dependency issue a known issue?

I am seeing the message (I am on Windows) and GitHub actions on Windows does see the same message.

LeakTest > stageLeakWithMenuButton() FAILED
    java.lang.AssertionError: The following references should be collected: [javafx.stage.Stage@b1712f3]

see https://github.com/danielpeintner/Java11Test/runs/3496191290?check_suite_focus=true#step:6:540

The test you've provided works on my machine. So the next step would be to look into the Heapdump and check the GC path.

I used VisualVM but I am not very experienced... I don't really understand why the AtomicReference is there twice and the value is null which should be fine I think.
Anyhow, thank you very much for your help. I don't wanna bother you more...

[FlorianKirmaier]

No, this issue is not known. Many those issues are not known - they are close to invisible without JMemoryBuddy.

To know what's happening, you will have to show the Path the "GC Root". There is a button for this in the upper right.
With an Image to the Path to the GC Root it's probably quite easy to understand what's happening - or event to fix it.

[danielpeintner]

With an Image to the Path to the GC Root it's probably quite easy to understand what's happening - or event to fix it.

I hope this is what you mean. To me it doesn't say a whole lot but maybe you can see the problem...

What surprises me is that there are 2 AtomicReference instances since there is only 1 in the test class.

[FlorianKirmaier]

The Atomic Reference is not really related to the leak. The two instances you are seeing, are internal implementation details of JavaFX.
You would do the following:

1. Search for "AssertCollectable"
2. Open it until you find the instance in the WeakReference, which caused the leak - and click on it.
3. Open the GC-Root area and look at the path to the GC Root.

[danielpeintner]

It seems to have to do with the combination of MenuButton and ControlAcceleratorSupport

I also tried to remove all accelerators from the scene but it did not help either
scene.getAccelerators().clear();

[danielpeintner]

I followed a related thread, see https://mail.openjdk.java.net/pipermail/openjfx-dev/2021-September/031925.html

Having said that, the test provided works with JavaFX 16 (also on Windows) but fails with version 17.