Incorrect project ID for secrets manager

[zjoyner-leageapps]

I have quarkus based application running in a GKE cluster in a project however when I look up a secret I know to be in the project the secret URL is for a different project. This will not work as the secrets are project specific. In this case they are DB passwords.

As a workaround I tried the fix in: #108
Namely, quarkus.google.cloud.access-token-enabled=false however that gave me a Null pointer with this stack trace:

java.lang.NullPointerException: Cannot read field "serviceAccountLocation" because "gcpConfiguration" is null at io.quarkiverse.googlecloudservices.common.GcpCredentialProducer.googleCredential(GcpCredentialProducer.java:46) at io.quarkiverse.googlecloudservices.common.GcpCredentialProducer_ProducerMethod_googleCredential_f50c1039bd62c63834aaa61a686a2c22210490cc_Bean.create(Unknown Source) at io.quarkiverse.googlecloudservices.common.GcpCredentialProducer_ProducerMethod_googleCredential_f50c1039bd62c63834aaa61a686a2c22210490cc_Bean.create(Unknown Source) at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:111) at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35) at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:32) at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:26) at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69) at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:32) at io.quarkiverse.googlecloudservices.common.GcpCredentialProducer_ProducerMethod_googleCredential_f50c1039bd62c63834aaa61a686a2c22210490cc_Bean.get(Unknown Source) at io.quarkiverse.googlecloudservices.common.GcpCredentialProducer_ProducerMethod_googleCredential_f50c1039bd62c63834aaa61a686a2c22210490cc_Bean.get(Unknown Source) at io.quarkiverse.googlecloudservices.secretmanager.runtime.SecretManagerProducer_Bean.create(Unknown Source) at io.quarkiverse.googlecloudservices.secretmanager.runtime.SecretManagerProducer_Bean.create(Unknown Source) at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:111) at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35) at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:32) at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:26) at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69) at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:32) at io.quarkiverse.googlecloudservices.secretmanager.runtime.SecretManagerProducer_Bean.get(Unknown Source) at io.quarkiverse.googlecloudservices.secretmanager.runtime.SecretManagerProducer_Bean.get(Unknown Source) at io.quarkiverse.googlecloudservices.secretmanager.runtime.SecretManagerProducer_ProducerMethod_secretManagerClient_da0d037e46ea741b638dc55997e8c2b5dd43f779_Bean.create(Unknown Source) at io.quarkiverse.googlecloudservices.secretmanager.runtime.SecretManagerProducer_ProducerMethod_secretManagerClient_da0d037e46ea741b638dc55997e8c2b5dd43f779_Bean.create(Unknown Source) at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:111) at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35) at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:32) at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:26) at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69) at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:32) at io.quarkiverse.googlecloudservices.secretmanager.runtime.SecretManagerProducer_ProducerMethod_secretManagerClient_da0d037e46ea741b638dc55997e8c2b5dd43f779_Bean.get(Unknown Source) at io.quarkiverse.googlecloudservices.secretmanager.runtime.SecretManagerProducer_ProducerMethod_secretManagerClient_da0d037e46ea741b638dc55997e8c2b5dd43f779_Bean.get(Unknown Source) at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:467) at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:480) at io.quarkus.arc.impl.ArcContainerImpl.instanceHandle(ArcContainerImpl.java:451) at io.quarkus.arc.impl.ArcContainerImpl.instance(ArcContainerImpl.java:244) at io.quarkiverse.googlecloudservices.secretmanager.runtime.config.SecretManagerClientProvider.get(SecretManagerClientProvider.java:18) at io.quarkiverse.googlecloudservices.secretmanager.runtime.config.SecretManagerConfigSource.getValue(SecretManagerConfigSource.java:46) at io.smallrye.config.ConfigValueConfigSourceWrapper.getConfigValue(ConfigValueConfigSourceWrapper.java:20) at io.smallrye.config.SmallRyeConfigSources.getValue(SmallRyeConfigSources.java:29) at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20) at io.smallrye.config.SecretKeysConfigSourceInterceptor.getValue(SecretKeysConfigSourceInterceptor.java:22) at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20) at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25) at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20) at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25) at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20) at io.smallrye.config.ProfileConfigSourceInterceptor.getValue(ProfileConfigSourceInterceptor.java:57) at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20) at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:44) at io.smallrye.config.ExpressionConfigSourceInterceptor.lambda$getValue$0(ExpressionConfigSourceInterceptor.java:57) at io.smallrye.common.expression.ExpressionNode.emit(ExpressionNode.java:22) at io.smallrye.common.expression.Expression.evaluateException(Expression.java:56) at io.smallrye.common.expression.Expression.evaluate(Expression.java:70) at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:56) at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:36) at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20) at io.smallrye.config.FallbackConfigSourceInterceptor.getValue(FallbackConfigSourceInterceptor.java:24) at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20) at io.smallrye.config.PropertyNamesConfigSourceInterceptor.getValue(PropertyNamesConfigSourceInterceptor.java:17) at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20) at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:307) at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:225) at io.quarkus.runtime.generated.Config.initGroup$io$quarkus$datasource$runtime$DataSourceRuntimeConfig(Unknown Source) at io.quarkus.runtime.generated.Config.rtGetEnclosing:datasource:DataSourcesRuntimeConfig#namedDataSources:*:DataSourceRuntimeConfig#password(Unknown Source) at io.quarkus.runtime.generated.Config.rtParseKey:quarkus:datasource:*:password(Unknown Source) at io.quarkus.runtime.generated.Config.rtParseKey:quarkus:datasource:*(Unknown Source) at io.quarkus.runtime.generated.Config.rtParseKey:quarkus:datasource(Unknown Source) at io.quarkus.runtime.generated.Config.rtParseKey:quarkus(Unknown Source) at io.quarkus.runtime.generated.Config.rtParseKey(Unknown Source) at io.quarkus.runtime.generated.Config.readConfig(Unknown Source) at io.quarkus.deployment.steps.RuntimeConfigSetup.deploy(Unknown Source) at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source) at io.quarkus.runtime.Application.start(Application.java:101) at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:103) at io.quarkus.runtime.Quarkus.run(Quarkus.java:67) at io.quarkus.runtime.Quarkus.run(Quarkus.java:41) at com.leagueapps.microservices.poc.potato.application.ApplicationStartup$Companion.main(ApplicationStartup.kt:19) at com.leagueapps.microservices.poc.potato.application.ApplicationStartup.main(ApplicationStartup.kt) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:60) at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:31)

If I apply the fix from: https://githubhot.com/repo/quarkiverse/quarkus-google-cloud-services/issues/209
That fixes the NPE but the application fails to start as it is still trying to get secrets from the other project.

This is my gradle dependancies:

dependencies {
	implementation("io.quarkiverse.googlecloudservices:quarkus-google-cloud-pubsub")
    implementation("io.quarkiverse.googlecloudservices:quarkus-google-cloud-secret-manager")
    implementation("io.quarkus:quarkus-micrometer-registry-prometheus")
    implementation("io.quarkus:quarkus-micrometer")
    implementation("io.quarkus:quarkus-container-image-jib")
    implementation("io.quarkus:quarkus-smallrye-health") {
		exclude("io.quarkus:quarkus-smallrye-jwt")
	}
	implementation("io.quarkus:quarkus-kubernetes")
	implementation(platform("com.google.cloud:libraries-bom:25.0.0"))
	implementation("com.google.cloud:google-cloud-secretmanager")
	modules {
		//The default is to use the shaded which does not have the features we need.
		module("io.grpc:grpc-netty-shaded") {
			replacedBy("io.grpc:grpc-netty")
		}
	}

	//implementation("io.swagger.core.v3:swagger-annotations:2.2.0")
	implementation("jakarta.annotation:jakarta.annotation-api:2.1.0")
	implementation("org.jetbrains.kotlin:kotlin-reflect")
	implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8")
	implementation("io.grpc:grpc-netty")
	runtimeOnly("ch.qos.logback:logback-classic:1.2.11")
	runtimeOnly("mysql:mysql-connector-java:8.0.29")
	runtimeOnly("com.fasterxml.jackson.module:jackson-module-kotlin:2.13.3")

	//exclude shaded netty, since it causes conflicts with graalvm
	configurations.implementation.get().exclude("io.grpc", "grpc-netty-shaded")

If you would like additional information please let me know.

[zjoyner-leageapps]

I added this code to confirm that the google API is returning the correct information:

			logger.info("[ApplicationStartup.main()] Entered method call.")
			System.setProperty("quarkus.google.cloud.project-id", ServiceOptions.getDefaultProjectId())
			logger.info("projectIDs: ${System.getProperty("quarkus.google.cloud.project-id")}")
			Quarkus.run(ApplicationInitializer::class.java, *args)
			logger.info("[ApplicationStartup.main()] Exiting method call.")

Which is displaying the correct project.

[zjoyner-leageapps]

I got things working but this issue is still present. The way I fixed it was to call getDefaultProjectID before starting the application and passing around a second property for the project name. And finally doing this in the application properties: ${sm//projects/${gc-project-id}/secrets/potato-poc-admin-password}

It looks like there was an attempted fix here: e7bbac9 But I don't think it worked.

[loicmathieu]

@zjoyner-leageapps which version of Quarkus and the io.quarkiverse.googlecloudservices:quarkus-google-cloud-secret-manager extension do you use ? The issue has been fixed in the latest version.

I also notice that you add a lot of dependencies that are managed by Quarkus and the Google Cloud extension pack, I strongly suggest to remove them to avoid possible dependency convergence issue :

implementation(platform("com.google.cloud:libraries-bom:25.0.0"))
	implementation("com.google.cloud:google-cloud-secretmanager")
	modules {
		//The default is to use the shaded which does not have the features we need.
		module("io.grpc:grpc-netty-shaded") {
			replacedBy("io.grpc:grpc-netty")
		}
	}
	implementation("io.grpc:grpc-netty")
	configurations.implementation.get().exclude("io.grpc", "grpc-netty-shaded")

There is also a Quarkus Kotlin support, you should use it, it'll be more easier to setup your application: https://quarkus.io/guides/kotlin

[zjoyner-leageapps]

This is the gradle deps relevant output:
io.quarkiverse.googlecloudservices:quarkus-google-cloud-secret-manager:1.1.1 -> 1.1.0
I am still working through why it isn't giving me the newest version but I so see some references to io.quarkiverse.googlecloudservices:quarkus-google-cloud-common:1.1.0 in the dependencies output which makes me curious.
(I dropped the first party google client for the moment as well.)

[zjoyner-leageapps]

Ok so I got it fail in a way that tracks with what I was seeing with my workaround by excluding secret-manager from the bom:
implementation("${quarkusPlatformGroupId}:quarkus-google-cloud-services-bom:${quarkusPlatformVersion}") { exclude(group="io.quarkiverse.googlecloudservices", module="quarkus-google-cloud-secret-manager") }
This tracks with the issue I was seeing with my workaround. It may be worth it to get the secrets manager updated in the bom.

[zjoyner-leageapps]

Ok so I got everything working. There were two issues:

• the 2.9.2 bom for quarkus-google-cloud-services-bom pins this project to 1.1.0 not 1.1.1 so we have to exclude it from the bom to set it in the gradle file as per the above.
• The other issue was with metadata.google.internal not resolving which meant that the projectID was not getting set. This isn't on your end but it may be worth it to make a note as I did not find anything out about in the google docs.

I am back online so feel free to close this if you want.

[loicmathieu]

@zjoyner-leageapps yes, the fix was made on 1.1.1 that should be included in Quarkus 2.10 I think (I'll check as I was on vacation and not sure I update the Quarkus Platform).

The other issue was with metadata.google.internal not resolving which meant that the projectID was not getting set. This isn't on your end but it may be worth it to make a note as I did not find anything out about in the google docs.

If you didn't set a projectID, the default is to use ServiceOptions.getDefaultProjectId() that will use the Google Metadata service to get it (it'll work on any GCP managed runtimes). It's documented on the documentation site: https://quarkiverse.github.io/quarkiverse-docs/quarkus-google-cloud-services/main/

Feel free to open a PR to enhance the documentation if you feel it must be more explicit ;)

I'm closing it now.

[loicmathieu]

I just checked and it would be OK in Quarkus 2.10.