Native Build with Quarkus 3.6.0 not working - java.lang.TypeNotPresentException: Type Lsoftware/amazon/awssdk/http/auth/aws/crt/internal/signer/DefaultAwsCrtV4aHttpSigner;

[manuelwallrapp]

Hi

when I do a native build with Quarkus 3.6.0 following exception appears:
java.lang.TypeNotPresentException: Type Lsoftware/amazon/awssdk/http/auth/aws/crt/internal/signer/DefaultAwsCrtV4aHttpSigner; not present

There seems to be something wrong since there exists a Delete_DefaultAwsCrtV4aHttpSigner.

[maven-build - mvn-goals] 2023-12-01T08:31:53.311491738Z ========================================================================================================================
[maven-build - mvn-goals] 2023-12-01T08:31:53.312557642Z GraalVM Native Image: Generating 'esta-tekton-pipeline-controller-2.23.8-SNAPSHOT-runner' (executable)...
[maven-build - mvn-goals] 2023-12-01T08:31:53.312909748Z ========================================================================================================================
[maven-build - mvn-goals] 2023-12-01T08:31:53.312940404Z For detailed information and explanations on the build output, visit:
[maven-build - mvn-goals] 2023-12-01T08:31:53.312955468Z https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
[maven-build - mvn-goals] 2023-12-01T08:31:53.312998400Z ------------------------------------------------------------------------------------------------------------------------
[maven-build - mvn-goals] 2023-12-01T08:31:55.009831824Z
[maven-build - mvn-goals] 2023-12-01T08:31:55.025326084Z [1/8] Initializing...                                                                                    (0.0s @ 0.46GB)
[maven-build - mvn-goals] 2023-12-01T08:31:55.055966528Z ------------------------------------------------------------------------------------------------------------------------
[maven-build - mvn-goals] 2023-12-01T08:31:55.061175631Z                         2.0s (12.8% of total time) in 29 GCs | Peak RSS: 1.42GB | CPU load: 3.17
[maven-build - mvn-goals] 2023-12-01T08:31:55.061694597Z ------------------------------------------------------------------------------------------------------------------------
[maven-build - mvn-goals] 2023-12-01T08:31:55.061710070Z Produced artifacts:
[maven-build - mvn-goals] 2023-12-01T08:31:55.063116501Z  /workspace/source/target/esta-tekton-pipeline-controller-2.23.8-SNAPSHOT-native-image-source-jar/svm_err_b_20231201T083155.040_pid3564.md (build_info)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063264473Z ========================================================================================================================
[maven-build - mvn-goals] 2023-12-01T08:31:55.063344639Z Failed generating 'esta-tekton-pipeline-controller-2.23.8-SNAPSHOT-runner' after 14.6s.
[maven-build - mvn-goals] 2023-12-01T08:31:55.063393268Z
[maven-build - mvn-goals] 2023-12-01T08:31:55.063404721Z The build process encountered an unexpected error:
[maven-build - mvn-goals] 2023-12-01T08:31:55.063430533Z
[maven-build - mvn-goals] 2023-12-01T08:31:55.063647401Z java.lang.TypeNotPresentException: Type Lsoftware/amazon/awssdk/http/auth/aws/crt/internal/signer/DefaultAwsCrtV4aHttpSigner; not present
[maven-build - mvn-goals] 2023-12-01T08:31:55.063658778Z 	at java.base/sun.reflect.annotation.TypeNotPresentExceptionProxy.generateException(TypeNotPresentExceptionProxy.java:47)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063668180Z 	at java.base/sun.reflect.annotation.AnnotationInvocationHandler.invoke(AnnotationInvocationHandler.java:89)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063668180Z 	at jdk.proxy3/jdk.proxy3.$Proxy16.value(Unknown Source)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063693206Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.substitute.AnnotationSubstitutionProcessor.findTargetClass(AnnotationSubstitutionProcessor.java:1032)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063693206Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.substitute.AnnotationSubstitutionProcessor.handleClass(AnnotationSubstitutionProcessor.java:373)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063704058Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.substitute.AnnotationSubstitutionProcessor.init(AnnotationSubstitutionProcessor.java:351)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063704058Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.createAnnotationSubstitutionProcessor(NativeImageGenerator.java:1030)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063712932Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.setupNativeImage(NativeImageGenerator.java:908)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063721354Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:591)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063744599Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:551)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063744599Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:538)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063759031Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:720)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063759031Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:142)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063768189Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:97)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063776603Z Caused by: java.lang.TypeNotPresentException: Type software.amazon.awssdk.http.auth.aws.crt.internal.signer.DefaultAwsCrtV4aHttpSigner not present
[maven-build - mvn-goals] 2023-12-01T08:31:55.063788744Z 	at java.base/sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:117)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063797951Z 	at java.base/sun.reflect.generics.visitor.Reifier.visitClassTypeSignature(Reifier.java:125)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063797951Z 	at java.base/sun.reflect.generics.tree.ClassTypeSignature.accept(ClassTypeSignature.java:49)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063808178Z 	at java.base/sun.reflect.annotation.AnnotationParser.parseSig(AnnotationParser.java:442)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063816767Z 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063827820Z 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063837042Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.annotation.AnnotationMetadata.extractType(AnnotationMetadata.java:73)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063846340Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.annotation.AnnotationClassValue.extract(AnnotationClassValue.java:39)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063855236Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.annotation.AnnotationMemberValue.extract(AnnotationMemberValue.java:44)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063863900Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.annotation.AnnotationValue.extract(AnnotationValue.java:66)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063874316Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.annotation.SubstrateAnnotationExtractor.lambda$getDeclaredAnnotationDataFromRoot$4(SubstrateAnnotationExtractor.java:252)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063882354Z 	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063889799Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.annotation.SubstrateAnnotationExtractor.getDeclaredAnnotationDataFromRoot(SubstrateAnnotationExtractor.java:242)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063897209Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.annotation.SubstrateAnnotationExtractor.lambda$getAnnotationDataFromRoot$3(SubstrateAnnotationExtractor.java:225)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063904605Z 	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1740)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063911983Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.annotation.SubstrateAnnotationExtractor.getAnnotationDataFromRoot(SubstrateAnnotationExtractor.java:224)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063919418Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.annotation.SubstrateAnnotationExtractor.getAnnotationData(SubstrateAnnotationExtractor.java:186)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063927755Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.annotation.SubstrateAnnotationExtractor.extractAnnotation(SubstrateAnnotationExtractor.java:124)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063936258Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.ImageClassLoader.handleClass(ImageClassLoader.java:177)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063945443Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageClassLoaderSupport$LoadClassHandler.handleClassFileName(NativeImageClassLoaderSupport.java:825)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063945443Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageClassLoaderSupport$LoadClassHandler$1.lambda$visitFile$0(NativeImageClassLoaderSupport.java:712)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063958909Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.ImageClassLoader$1.lambda$execute$0(ImageClassLoader.java:94)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063968175Z 	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063968175Z 	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063978912Z 	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063988332Z 	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
[maven-build - mvn-goals] 2023-12-01T08:31:55.063996808Z 	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
[maven-build - mvn-goals] 2023-12-01T08:31:55.064006740Z 	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
[maven-build - mvn-goals] 2023-12-01T08:31:55.064049763Z Caused by: java.lang.ClassNotFoundException: software.amazon.awssdk.http.auth.aws.crt.internal.signer.DefaultAwsCrtV4aHttpSigner
[maven-build - mvn-goals] 2023-12-01T08:31:55.064082234Z 	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
[maven-build - mvn-goals] 2023-12-01T08:31:55.064109640Z 	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
[maven-build - mvn-goals] 2023-12-01T08:31:55.064109640Z 	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
[maven-build - mvn-goals] 2023-12-01T08:31:55.064118753Z 	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageClassLoader.loadClass(NativeImageClassLoader.java:652)
[maven-build - mvn-goals] 2023-12-01T08:31:55.064124920Z 	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
[maven-build - mvn-goals] 2023-12-01T08:31:55.064132360Z 	at java.base/java.lang.Class.forName0(Native Method)
[maven-build - mvn-goals] 2023-12-01T08:31:55.064138440Z 	at java.base/java.lang.Class.forName(Class.java:534)
[maven-build - mvn-goals] 2023-12-01T08:31:55.064144466Z 	at java.base/java.lang.Class.forName(Class.java:513)
[maven-build - mvn-goals] 2023-12-01T08:31:55.064152266Z 	at java.base/sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:114)
[maven-build - mvn-goals] 2023-12-01T08:31:55.064160901Z 	... 27 more

[armujahid]

Noticed this after upgrading quarkus-amazon-services-bom from 2.5.2 to 2.5.3

[scrocquesel]

Given the log, I would say you are using an old version of GraalVM.

Could you give a bit more information on the GraalVM you use ?
The builld should pass with quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-21 which has Java version: 21.0.1+12-LTS, vendor version: Mandrel-23.1.1.0-Final

Recent version usually give their version in the Initializing phase like

2023-12-01T19:40:39.5790641Z ========================================================================================================================
2023-12-01T19:40:39.5802615Z GraalVM Native Image: Generating 'quarkus-amazon-services-integration-tests-999-SNAPSHOT-runner' (executable)...
2023-12-01T19:40:39.5803781Z ========================================================================================================================
2023-12-01T19:40:39.5804706Z For detailed information and explanations on the build output, visit:
2023-12-01T19:40:39.5805710Z https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
2023-12-01T19:40:39.5806656Z ------------------------------------------------------------------------------------------------------------------------
2023-12-01T19:40:43.0316369Z [1/8] Initializing...                                                                                    (8.6s @ 0.27GB)
2023-12-01T19:40:43.0320224Z  Java version: 21.0.1+12-LTS, vendor version: Mandrel-23.1.1.0-Final
2023-12-01T19:40:43.0334671Z  Graal compiler: optimization level: 2, target machine: x86-64-v3
2023-12-01T19:40:43.0335553Z  C compiler: gcc (redhat, x86_64, 8.5.0)
2023-12-01T19:40:43.0341430Z  Garbage collector: Serial GC (max heap size: 80% of RAM)

See #857 (comment) for tips to keep your ci/cd jobs up to date.

[manuelwallrapp]

Yes exactly, it is also my impression it happened after the version bump. My Graal VM is this:

java 21.0.1 2023-10-17
Java(TM) SE Runtime Environment Oracle GraalVM 21.0.1+12.1 (build 21.0.1+12-jvmci-23.1-b19)
Java HotSpot(TM) 64-Bit Server VM Oracle GraalVM 21.0.1+12.1 (build 21.0.1+12-jvmci-23.1-b19, mixed mode, sharing)

[scrocquesel]

I'm a bit clueless. I tried building our integration tests with GraalVM instead of Mandrel and it passed.

mvn clean package -Dquarkus.package.type=native -Dquarkus.native.container-build=true -DskipTests -Dquarkus.native.builder-image=graalvm

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on GRAALVM 23.1 JDK 21.0.1+12-jvmci-23.1-b19
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] docker run --env LANG=C --rm --user 1000:1000 -v /home/cortex/source/quarkus/quarkus-amazon-services/integration-tests/target/quarkus-amazon-services-integration-tests-999-SNAPSHOT-native-image-source-jar:/project:z --name build-native-WFkXy quay.io/quarkus/ubi-quarkus-graalvmce-builder-image:jdk-21 -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dlogging.initial-configurator.min-level=500 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Dfile.encoding=UTF-8 --features=io.quarkus.runner.Feature,io.quarkus.runtime.graal.DisableLoggingFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:+UnlockExperimentalVMOptions -H:BuildOutputJSONFile=quarkus-amazon-services-integration-tests-999-SNAPSHOT-runner-build-output-stats.json -H:-UnlockExperimentalVMOptions --strict-image-heap -H:+UnlockExperimentalVMOptions -H:+AllowFoldMethods -H:-UnlockExperimentalVMOptions -J-Djava.awt.headless=true --no-fallback --link-at-build-time -H:+UnlockExperimentalVMOptions -H:+ReportExceptionStackTraces -H:-UnlockExperimentalVMOptions -H:-AddAllCharsets --enable-url-protocols=http,https --enable-monitoring=heapdump -H:+UnlockExperimentalVMOptions -H:-UseServiceLoaderFeature -H:-UnlockExperimentalVMOptions -J--add-exports=org.graalvm.nativeimage/org.graalvm.nativeimage.impl=ALL-UNNAMED --exclude-config io\.netty\.netty-codec /META-INF/native-image/io\.netty/netty-codec/generated/handlers/reflect-config\.json --exclude-config io\.netty\.netty-handler /META-INF/native-image/io\.netty/netty-handler/generated/handlers/reflect-config\.json quarkus-amazon-services-integration-tests-999-SNAPSHOT-runner -jar quarkus-amazon-services-integration-tests-999-SNAPSHOT-runner.jar
Warning: Using a deprecated option --allow-incomplete-classpath from 'META-INF/native-image/software.amazon.awssdk/netty-nio-client/native-image.properties' in 'file:///project/lib/software.amazon.awssdk.netty-nio-client-2.21.29.jar'. Allowing an incomplete classpath is now the default. Use --link-at-build-time to report linking errors at image build time for a class or package.
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/io.netty/netty-transport/reflection-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: Please re-evaluate whether any experimental option is required, and either remove or unlock it. The build output lists all active experimental options, including where they come from and possible alternatives. If you think an experimental option should be considered as stable, please file an issue.
========================================================================================================================
GraalVM Native Image: Generating 'quarkus-amazon-services-integration-tests-999-SNAPSHOT-runner' (executable)...
========================================================================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
------------------------------------------------------------------------------------------------------------------------
[1/8] Initializing...                                                                                    (8.5s @ 0.29GB)
 Java version: 21.0.1+12, vendor version: GraalVM CE 21.0.1+12.1
 Graal compiler: optimization level: 2, target machine: x86-64-v3
 C compiler: gcc (redhat, x86_64, 8.5.0)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 3 user-specific feature(s):

Could you build your project with the same command line ? If you can share a small reproducer or pom.xml

[manuelwallrapp]

Ok, thank you for testing @scrocquesel. I digged deeper in our Maven Dependency tree and found an issue with the versions which I could solve.