Making SSH Nodes work both on Linux and Windows based systems

[dmatej]

Please try to review "per commit" - it is a lot of changes, tested on Kubuntu host and Windows Server 2025 in VirtualBox. I know it works, however I could make some mistakes.

When I implemented these tests I had to face quite a lot of issues even on Linux. When I finish, I will write some blog about it, it is another painful episode. However I have to make notes and it will be useful also for users and reviewers.

Automated test

• Expects you can use local docker service on Linux.

Manual test

• The private key doesn't have any passphrase
• Test uses Temurin docker images as a base.
• You have Windows Server 2025 installed into VirtualBox machine, reachable using network and configured to provide the SSH Server with SFTP support.
• Later, tired of repeating actions, I created two scripts.

Windows Server 2025 Configuration

I hope I did not skip something, but I tried to collect command line executions done to prepare everything. It is a bit longer, so I will add that to discussions, see #25343

DAS Script

#!/bin/bash
set -x

VERSION="$1"
./glassfish7/bin/asadmin stop-domain
rm -rf ./glassfish7

set -e
mvn org.apache.maven.plugins:maven-dependency-plugin:3.1.2:get -Dartifact=org.glassfish.main.distributions:glassfish:${VERSION}:zip -Dtransitive=false
cp ${HOME}/.m2/repository/org/glassfish/main/distributions/glassfish/${VERSION}/glassfish-${VERSION}.zip ./
unzip glassfish-${VERSION}.zip
#cp ./asenv.conf ./glassfish7/glassfish/config/asenv.conf
export AS_JAVA=/usr/lib/jvm/jdk17
./glassfish7/bin/asadmin start-domain --debug
tail -n 10000 -F ./glassfish7/glassfish/domains/domain1/logs/server.log

Test Script

#!/bin/bash

set -v
set -e

#/app/appservers/glassfish7/bin/asadmin start-domain

/app/appservers/glassfish7/bin/asadmin change-admin-password
/app/appservers/glassfish7/bin/asadmin enable-secure-admin
/app/appservers/glassfish7/bin/asadmin restart-domain

export AS_ADMIN_PASSWORDFILE=/app/appservers/passwordfile.txt
/app/appservers/glassfish7/bin/asadmin create-node-ssh --nodehost 192.168.56.101 --installdir '/Users/vboxuser/glassfish7' --sshuser vboxuser --sshkeyfile /home/dmatej/.ssh/id_rsa_nopassphrase --install true win2025
/app/appservers/glassfish7/bin/asadmin create-cluster ClusterOne
/app/appservers/glassfish7/bin/asadmin create-instance --cluster ClusterOne --node win2025 InstanceWin2025
/app/appservers/glassfish7/bin/asadmin create-instance --cluster ClusterOne --node localhost-domain1 InstanceLin2025
#/app/appservers/glassfish7/bin/asadmin start-cluster ClusterOne

/app/appservers/glassfish7/bin/asadmin start-instance InstanceWin2025
/app/appservers/glassfish7/bin/asadmin list-instances 

The passwordfile.txt contains just this:

AS_ADMIN_PASSWORD=admin123

Execution

You need three console windows:
Window 1: cd glassfish project, clone from github, and build and run automated tests:

mvn clean install -Pfastest -T4C && mvn clean install -pl :ssh-cluster-tests

Window 2, start DAS and print the server.log:

./run-jdk17-gf7.sh 7.0.22-SNAPSHOT

Window 3, execute the test scenario. It will ask three times for password, which is admin123

./run.sh 

Node side: Environment

• jar is not on PATH everywhere
  • however unzip might be already there - fix: add some autodetection code.

Node side: SSHD Configuration

• SFTP has to be explicitly enabled on some systems
• public key authentication can be disabled. Same applies to other types.
• User interaction can be forbidden.
• Port can be blocked by firewall or antivirus, etc.
• Port can be different.
• ...

DAS side: Environment

• When using scripts, no TTY might be available. Scripts can fail or wait for user input forever.
  • Always set timeout when such risk exists
  • For ssh command is useful sshpass command. Test uses it, however GlassFish uses Jsch library instead.

DAS side: SSH Command Configuration

• Applies to the test and user experiments, not to GlassFish
• Empty passphrase - be careful, when you set it via execInContainer: "''" means apostrophes, not empty passphrase.
• User interaction can be forbidden just like on the server (sshd)
• StrictHostKeyChecking=accept-new should not be used on production; however would be useful if we would not enforce user to run the ssh command on his own and manually accept the server's key. That is TODO for another PR, however I replaced all none values by accept-new, because it is still better than not doing any validation. User can do the validation manually first, than DAS does not override it.

DAS side: JSCH SSH Implementation

• "raw" and "resolved" password and passphrase. I doubt the implementation is correct.
• logging
  • race condition - fix: use Jsch logger name.
  • impossible to guess which logger level I should set, overengineered antipattern of "set logger for Jsch once for asadmin, then for kernel, then for cluster, then just jsch, ..." - then you set 4 logger levels and it still doesn't print anything. The fix: always use Jsch logger name.

[dmatej]

Local status: Large refactoring targeting File vs Path vs String vs SFTP vs Unix vs Windows. Local manual testing of DAS on Kubuntu and Node on Windows Server 2025 - was able to "replicate" glassfish installation, unpack, create node and instance. Instance did not start yet as I did not install Java to Windows yet :-)

TODO

• Proper session management (AutoCloseable, separate session and SSHLauncher)
• Data object with some autodetection of Node's operating system and features
• SFTP rmdir does not work on windows
• Documentation, example, ...

[dmatej]

TODO:

• Broken encoding in SSH stdout (hardcoded UTF8 vs Windows-1250)
• Missing autodeploy/bundles (I am often removing those empty directories with .empty file inside, this time it seems Felix cannot autocreate it)
• restart-domain now does not replace WALL_CLOCK_START system property, then GF computes incorrect startup time including CLI (now rather "time since CLI started this process"; on restart-domain the server restarts itself after it was asked to do so by restart-domain).
• SSH adapter should throw the new SSHException, which would always collect exec output. We are losing remote logs sometimes, details about errors. I have fixed it here and there where I needed it, but this should be systematic.

[dmatej]

Yet one bug detected (I created it): NIO Path uses '' as a separator, not '/', so it fails to use SFTP protocol now.