Cannot create a user or add kubernetes cluster as specified in the Getting started document.

[s-bethi]

Hi, I am trying to use apollo to push deployments to kubernetes cluster.

I am following the steps specified in the Getting started document
https://github.com/logzio/apollo/wiki/Getting-Started-with-Apollo

I brought up the docker containers using docker-compose up -d from apollo/examples directory.

Both the mysql container examples_db_1 and apollo examples_apollo_1are up. I am able to generate a token for default user as mentioned in the document. But i am not able to create any new user or add a kubernetes cluster, I am getting 404 errors in both cases.

The logs of example_apollo_1 container are below, i see errors and i am not sure what i am missing.

[main] INFO jetty - using temp directory for jetty: /root/.hawtio/tmp
Embedded hawtio: You can use --help to show usage
Using options [
        war=/tmp/hawtio-3115128339803883963.war
        contextPath=/hawtio
        port=8083
        plugins=plugins
        openUrl=true
        jointServerThread=false
        help=false]
About to start hawtio /tmp/hawtio-3115128339803883963.war
[main] INFO org.eclipse.jetty.server.Server - jetty-8.y.z-SNAPSHOT
[main] INFO org.eclipse.jetty.webapp.WebInfConfiguration - Extract jar:file:/tmp/hawtio-3115128339803883963.war!/ to /root/.hawtio/tmp/webapp
20/07/14 17:10:02.698 [main] INFO i.l.a.ApolloApplication: Starting apollo..
20/07/14 17:10:02.735 [main] WARN c.n.g.l.ClasspathScanner: No base packages specified - no classpath scanning will be done
20/07/14 17:10:03.029 [main] INFO c.n.g.LifecycleManager: Starting 'LifecycleManager@811597470'
20/07/14 17:10:03.045 [main] INFO c.n.g.g.ModuleListBuilder: Adding module 'io.logz.apollo.di.ApolloModule
20/07/14 17:10:03.045 [main] INFO c.n.g.g.ModuleListBuilder: Adding module 'io.logz.apollo.di.ApolloMyBatisModule
20/07/14 17:10:03.260 [main] INFO o.r.Reflections: Reflections took 191 ms to scan 1 urls, producing 27 keys and 156 values
20/07/14 17:10:03.314 [main] INFO o.f.c.i.u.VersionPrinter: Flyway 4.2.0 by Boxfuse
Tue Jul 14 17:10:03 GMT 2020 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
20/07/14 17:10:03.851 [main] INFO o.f.c.i.d.DbSupportFactory: Database: jdbc:mysql://db.local:3306/apollo?createDatabaseIfNotExist=true (MySQL 5.7)
20/07/14 17:10:03.942 [main] INFO o.f.c.i.c.DbValidate: Successfully validated 30 migrations (execution time 00:00.019s)
20/07/14 17:10:03.963 [main] INFO o.f.c.i.m.MetaDataTableImpl: Creating Metadata table: `apollo`.`schema_version`
[main] INFO org.eclipse.jetty.webapp.StandardDescriptorProcessor - NO JSP Support for /hawtio, did not find org.apache.jasper.servlet.JspServlet
20/07/14 17:10:04.052 [main] INFO o.f.c.i.c.DbMigrate: Current version of schema `apollo`: << Empty Schema >>
20/07/14 17:10:04.052 [main] WARN o.f.c.i.c.DbMigrate: outOfOrder mode is active. Migration of schema `apollo` may not be reproducible.
20/07/14 17:10:04.069 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 1 - Apollo baseline
[main] INFO io.hawt.system.ConfigManager - Configuration will be discovered via system properties
[main] INFO io.hawt.jmx.JmxTreeWatcher - Welcome to hawtio 1.4.67 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
[main] INFO io.hawt.jmx.UploadManager - Using file upload directory: /tmp/uploads
[main] INFO /hawtio - Loading Blueprint contexts [file:/root/.hawtio/tmp/webapp/WEB-INF/classes/OSGI-INF/blueprint/blueprint.xml, jar:file:/root/.hawtio/tmp/webapp/WEB-INF/lib/hawtio-aether-1.4.67.jar!/OSGI-INF/blueprint/blueprint.xml, jar:file:/root/.hawtio/tmp/webapp/WEB-INF/lib/hawtio-core-1.4.67.jar!/OSGI-INF/blueprint/blueprint.xml, jar:file:/root/.hawtio/tmp/webapp/WEB-INF/lib/hawtio-git-1.4.67.jar!/OSGI-INF/blueprint/blueprint.xml, jar:file:/root/.hawtio/tmp/webapp/WEB-INF/lib/hawtio-ide-1.4.67.jar!/OSGI-INF/blueprint/blueprint.xml, jar:file:/root/.hawtio/tmp/webapp/WEB-INF/lib/hawtio-json-schema-mbean-1.4.67.jar!/OSGI-INF/blueprint/blueprint.xml, jar:file:/root/.hawtio/tmp/webapp/WEB-INF/lib/hawtio-local-jvm-mbean-1.4.67.jar!/OSGI-INF/blueprint/blueprint.xml]
20/07/14 17:10:04.231 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20170530112639 - Add default shell
20/07/14 17:10:04.276 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20170604113058 - Blockers
20/07/14 17:10:04.305 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20170723163453 - Commit message x 10
20/07/14 17:10:04.324 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20170829110156 - Add notifications
20/07/14 17:10:04.353 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20171112150325 - Add services status column
20/07/14 17:10:04.395 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20171130120411 - Add table group
20/07/14 17:10:04.426 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20171130121344 - Add is part of group column
20/07/14 17:10:04.461 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20171204113102 - Add columns to deployment table
20/07/14 17:10:04.533 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20180109152101 - Add column to groups table
[main] INFO io.hawt.log.log4j.Log4jLogQuery - Connected to Log4j appender to trap logs with hawtio log plugin
20/07/14 17:10:04.570 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20180117130344 - Add default admin user
[main] INFO io.hawt.git.GitFacade - hawtio using config directory: /root/.hawtio/config
[main] INFO io.hawt.git.GitFacade - Cloning git repo https://github.com/hawtio/hawtio-config.git into directory /root/.hawtio/config cloneAllBranches: false
20/07/14 17:10:04.581 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20180117165543 - Refactor permissions tables names
20/07/14 17:10:04.612 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20180118114433 - Refactor columns names in permissions tables
20/07/14 17:10:04.780 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20180121003411 - Not null is part of group column
20/07/14 17:10:04.816 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20180215122723 - User blocker override
20/07/14 17:10:04.841 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20180225211644 - Deployment message
20/07/14 17:10:04.909 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20180306163755 - Add columns to environments
20/07/14 17:10:04.943 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20180319011255 - Add service ingress column to services
20/07/14 17:10:04.979 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20180530104010 - Add disabled users
20/07/14 17:10:05.043 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20180627150848 - Add concurrency limit
20/07/14 17:10:05.078 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20181223184648 - Add stacks
20/07/14 17:10:05.142 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20190123105216 - Enlarge token field in environment table
20/07/14 17:10:05.176 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20190130115519 - Add column to environment table
20/07/14 17:10:05.209 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20190131141930 - Add ca cert column to environment table
20/07/14 17:10:05.247 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20190613181915 - Add column to deployment
20/07/14 17:10:05.297 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20190619113055 - Add column to blocker definition
20/07/14 17:10:05.333 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20191104163332 - Add another column to blocker definition
20/07/14 17:10:05.369 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20191225164134 - Add column to users
20/07/14 17:10:05.429 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20200309182111 - Add slaves
20/07/14 17:10:05.454 [main] INFO o.f.c.i.c.DbMigrate: Migrating schema `apollo` to version 20200520101837 - Add active column to environments
20/07/14 17:10:05.486 [main] INFO o.f.c.i.c.DbMigrate: Successfully applied 30 migrations to schema `apollo` (execution time 00:01.524s).
20/07/14 17:10:05.497 [main] INFO i.l.a.d.DataSourceFactory: Creating connection pool with these parameters: {dataSource.databaseName=apollo, dataSource.serverName=db.local, registerMbeans=true, poolName=apollo, dataSourceClassName=org.mariadb.jdbc.MariaDbDataSource, username=root, maximumPoolSize=50, minimumIdle=1, connectionTimeout=5000, password=rootpassword, dataSource.port=3306}, {port=3306, properties=useUnicode=true;characterEncoding=UTF-8, databaseName=apollo, serverName=db.local}
20/07/14 17:10:05.500 [main] INFO c.z.h.HikariDataSource: apollo - Started.
[main] WARN io.hawt.jvm.local.JVMList - Local JVM discovery disabled as this JVM cannot access com.sun.tools.attach.VirtualMachine due to: com/sun/tools/attach/VirtualMachine
[main] INFO io.hawt.web.AuthenticationFilter - Starting hawtio authentication filter, JAAS authentication disabled
[main] INFO /hawtio - jolokia-agent: Using policy access restrictor classpath:/jolokia-access.xml
[main] INFO org.eclipse.jetty.webapp.WebAppContext - hawtio at http://0.0.0.0:8083/hawtio
[main] INFO org.eclipse.jetty.server.AbstractConnector - Started [email protected]:8083

hawtio: Don't cha wish your console was hawt like me!
=====================================================

http://localhost:8083/hawtio

20/07/14 17:10:06.344 [main] INFO o.r.c.RapidoidInitializer: Starting Rapidoid v5.3.4, built on 2017-04-02 23:41 UTC
20/07/14 17:10:06.345 [main] INFO o.r.c.RapidoidInitializer: System info | os = Linux | java = 1.8.0_111-internal | process = 11@d53bed9e8893 | max memory = 14313 MB | dir = /
20/07/14 17:10:06.353 [main] INFO o.r.e.Environment: No profiles were specified, activating 'default' profile
20/07/14 17:10:06.358 [main] INFO o.r.e.Environment: No production/dev/test mode was configured, inferring mode | mode = PRODUCTION
20/07/14 17:10:06.359 [main] INFO o.r.e.Environment: Automatically activating mode-specific profile | profile = production
20/07/14 17:10:06.359 [main] INFO o.r.e.Environment: Initialized environment | mode = PRODUCTION | profiles = [default, production]
20/07/14 17:10:06.502 [main] INFO o.r.c.ConfigImpl: Loaded configuration | namespace = config | files = [built-in-config.yml, built-in-config-default.yml]
20/07/14 17:10:06.567 [main] INFO o.r.s.App: Inferring application root | main = io.logz.apollo.ApolloApplication | package = io.logz.apollo.rest
20/07/14 17:10:06.584 [server] INFO o.r.n.i.RapidoidServerLoop: Starting server | address = 0.0.0.0 | port = 8081 | I/O workers = 8 | sync = true | accept = non-blocking
20/07/14 17:10:06.993 [main] INFO o.r.s.Setup: Server has started | setup = app | home = http://localhost:8081
20/07/14 17:10:06.994 [main] INFO o.r.s.Setup: Static resources will be served from the following locations | setup = app | locations = [static, default/static]
20/07/14 17:10:07.001 [main] INFO o.r.h.i.HttpRoutesImpl: POST /_login                                   | setup = app | roles = [] | transaction = NONE | mvc = false | cacheTTL = 0
20/07/14 17:10:07.006 [main] INFO o.r.h.i.HttpRoutesImpl: GET /_logout                                   | setup = app | roles = [logged_in] | transaction = NONE | mvc = false | cacheTTL = 0
Exception in thread "main" java.lang.Error: java.lang.reflect.InvocationTargetException: invokedynamic: method=public void io.logz.apollo.rest.RestServer.start(), target=io.logz.apollo.rest.RestServer@5f462e3b
        at com.netflix.governator.guice.InternalLifecycleModule.onProvision(InternalLifecycleModule.java:88)
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126)
        at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:92)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
        at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
        at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
        at com.google.inject.internal.InjectorImpl.createChildInjector(InjectorImpl.java:232)
        at com.netflix.governator.guice.LifecycleInjector.createChildInjector(LifecycleInjector.java:331)
        at com.netflix.governator.guice.LifecycleInjector.createInjector(LifecycleInjector.java:411)
        at com.netflix.governator.guice.LifecycleInjector.createInjector(LifecycleInjector.java:352)
        at io.logz.apollo.ApolloApplication.start(ApolloApplication.java:37)
        at io.logz.apollo.ApolloApplication.main(ApolloApplication.java:70)
Caused by: java.lang.reflect.InvocationTargetException: invokedynamic: method=public void io.logz.apollo.rest.RestServer.start(), target=io.logz.apollo.rest.RestServer@5f462e3b
        at com.netflix.governator.lifecycle.LifecycleMethods.methodInvoke(LifecycleMethods.java:316)
        at com.netflix.governator.lifecycle.LifecycleMethods.methodInvoke(LifecycleMethods.java:298)
        at com.netflix.governator.lifecycle.LifecycleManager.startInstance(LifecycleManager.java:266)
        at com.netflix.governator.lifecycle.LifecycleManager.add(LifecycleManager.java:175)
        at com.netflix.governator.guice.InternalLifecycleModule.onProvision(InternalLifecycleModule.java:81)
        ... 21 more
Caused by: com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Error injecting constructor, java.lang.NullPointerException
  at io.logz.apollo.services.SlaveService.<init>(SlaveService.java:45)
  at io.logz.apollo.di.ApolloModule.configure(ApolloModule.java:39)
  while locating io.logz.apollo.services.SlaveService
    for the 8th parameter of io.logz.apollo.kubernetes.KubernetesMonitor.<init>(KubernetesMonitor.java:57)
  at io.logz.apollo.di.ApolloModule.configure(ApolloModule.java:32)
  while locating io.logz.apollo.kubernetes.KubernetesMonitor

1 error
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1028)
        at org.rapidoid.integrate.GuiceBeans.getBeans(GuiceBeans.java:85)
        at org.rapidoid.integrate.GuiceBeans.getAnnotated(GuiceBeans.java:60)
        at org.rapidoid.setup.Setup.register(Setup.java:591)
        at org.rapidoid.setup.App.register(App.java:285)
        at io.logz.apollo.rest.RestServer.start(RestServer.java:65)
        at com.netflix.governator.lifecycle.LifecycleMethods.methodInvoke(LifecycleMethods.java:313)
        ... 25 more
Caused by: java.lang.NullPointerException
        at io.logz.apollo.services.SlaveService.<init>(SlaveService.java:50)
        at io.logz.apollo.services.SlaveService$$FastClassByGuice$$79991b1f.newInstance(<generated>)
        at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111)
        at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:33)
        at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:95)
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115)
        at com.netflix.governator.LifecycleListenerModule$LifecycleListenerProvisionListener.onProvision(LifecycleListenerModule.java:51)
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126)
        at com.netflix.governator.guice.InternalLifecycleModule.onProvision(InternalLifecycleModule.java:65)
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126)
        at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:92)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
        at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:33)
        at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:95)
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115)
        at com.netflix.governator.LifecycleListenerModule$LifecycleListenerProvisionListener.onProvision(LifecycleListenerModule.java:51)
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126)
        at com.netflix.governator.guice.InternalLifecycleModule.onProvision(InternalLifecycleModule.java:65)
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126)
        at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68)
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:92)
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
        ... 31 more```

 

[roiravhon]

Hi @s-bethi, thanks for reporting!
We think that the issue may originate in a new feature that was added without reflecting in the sample config. Can you pull master and try again?

[s-bethi]

Thank you @roiravhon for getting back to me.
i did a git pull and now i am able to make it work. I am able to create users, add environment and deployment permissions using API as stated in the Getting started document.

However i am not able to deploy the sample-apollo-app to kubernetes cluster.
I am not able to see the environment in Apollo UI
when i click on "New Deployment" it directly takes me to the following screen and when i hit next it doesn't do/show anything.

shouldn't i see the environment as specified in the Getting started document?

[172.19.0.19] $ curl -X POST http://localhost:8081/signup?_token=$token \
>   -H 'content-type: application/json' \
>   -d '{
> "firstName": "sharath",
> "lastName": "bethi",
> "userEmail": "[email protected]",
> "password": "password"
> }'
{"userEmail":"[email protected]","firstName":"sharath","lastName":"bethi","hashedPassword":"1f2eaef582ce135e9a3dbf172b48b667","enabled":true,"admin":false,"execAllowed":true}[Sun Jul 19 14:45:54] root@loqa19:/home/admin/apollo
[172.19.0.19] $

[172.19.0.19] $ curl -X PUT http://localhost:8081/users?_token=$token \
>   -H 'content-type: application/json' \
>   -d '{
> "firstName": "sharath",
> "lastName": "bethi",
> "userEmail": "[email protected]",
> "password": "password",
>         "isEnabled": true,
> "isAdmin": true
> }'
{"userEmail":"[email protected]","firstName":"sharath","lastName":"bethi","hashedPassword":"1f2eaef582ce135e9a3dbf172b48b667","enabled":true,"admin":true,"execAllowed":true}

 curl -X POST http://localhost:8081/environment?_token=$token1 \
>   -H 'content-type: application/json' \
>   -d '{
> "name": "test",
> "geoRegion": "us-east-1",
> "availability": "staging",
> "kubernetesMaster": "http://172.19.0.19:8080",
> "kubernetesNamespace": "default",
> "kubernetesToken": "$SECRET_TOKEN",
> "servicePortCoefficient": 0
> }'
{"id":1,"name":"test","geoRegion":"us-east-1","availability":"staging","kubernetesMaster":"http://172.19.0.19:8080","kubernetesToken":"$SECRET_TOKEN","kubernetesCaCert":null,"kubernetesNamespace":"default","servicePortCoefficient":0,"requireDeploymentMessage":null,"requireHealthCheck":null,"concurrencyLimit":null,"additionalParams":null,"isActive":null}

curl -X POST http://localhost:8081/deployment-roles?_token=$token1 \
>   -H 'content-type: application/json' \
>   -d '{
> "name": "devs"
> }'
{"id":1,"name":"devs"}

curl -X POST http://localhost:8081/deployment-roles/add-user?_token=$token1 \
>   -H 'content-type: application/json' \
>   -d '{
> "userEmail": "[email protected]",
> "deploymentRoleId": "1"
> }'
"ok"

[172.19.0.19] $ curl -X POST http://localhost:8081/deployment-permission?_token=$token1 \
>   -H 'content-type: application/json' \
>   -d '{
> "name": "Nginx allow",
> "serviceId": "1",
> "environmentId": "1",
> "permissionType": "ALLOW"
> }'
{"id":1,"name":"Nginx allow","serviceId":1,"environmentId":1,"permissionType":"ALLOW"}

curl -X POST http://localhost:8081/deployment-roles/add-deployment-permission?_token=$token1 \
>   -H 'content-type: application/json' \
>   -d '{
> "deploymentRoleId": 1,
> "deploymentPermissionId": 1
> }'
"ok"

[172.19.0.19] $ curl -X POST http://localhost:8081/deployable-version?_token=$token1 \
>   -H 'content-type: application/json' \
>   -d '{
> "gitCommitSha": "df821c0ce4dbacef9c5c79ccf0d4b1a59d150568",
> "githubRepositoryUrl": "https://github.com/logzio/sample-apollo-app",
> "serviceId": "1"
> }'
{"id":1,"gitCommitSha":"df821c0ce4dbacef9c5c79ccf0d4b1a59d150568","githubRepositoryUrl":"https://github.com/logzio/sample-apollo-app","serviceId":1,"commitUrl":"https://github.com/logzio/sample-apollo-app/commit/df821c0ce4dbacef9c5c79ccf0d4b1a59d150568","commitMessage":"Create LICENSE","commitDate":1516267277000,"committerAvatarUrl":"https://avatars0.githubusercontent.com/u/8061130?v=4","committerName":"Roi Rav-Hon"}

[roiravhon]

We need to update that guide :)
Can you try to edit the environment (either via API or directly in the DB) to have the "isActive" flag set to 1?

[s-bethi]

Thank you again, i set the value to 1 and now i am able to deploy the sample app to the cluster.
I am able to get the logs of the pod, restart the pod however i am not able to exec into the pod it is stuck here

I am not sure what i am missing.
here are the logs from apollo docker container

20/07/20 16:35:24.254 [qtp1236303587-532] INFO i.l.a.w.e.AuthenticationFilter: Granted Live-Session permission to user [email protected] on service 1 and environment 1
20/07/20 16:35:24.259 [qtp1236303587-532] INFO i.l.a.w.e.ContainerExecEndpoint: Opening ExecWatch to container sample-apollo-app in pod sample-apollo-app-654f5755d8-j79xx in environment test related to service sample-apollo-app
20/07/20 16:35:24.267 [OkHttp http://172.19.0.19:8080/...] ERROR i.f.k.c.d.i.ExecWebSocketListener: Exec Failure: HTTP:403. Message:Forbidden

java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
        at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229)
        at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196)
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
20/07/20 16:35:24.268 [qtp1236303587-532] WARN o.e.j.w.j.a.JsrEvents: Unable to report throwable to websocket (no @OnError handler declared): io.logz.apollo.websockets.exec.ContainerExecEndpoint
io.fabric8.kubernetes.client.KubernetesClientException: Forbidden

        at io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener.onFailure(ExecWebSocketListener.java:237)
        at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
        at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
20/07/20 16:35:24.268 [qtp1236303587-532] WARN o.e.j.w.c.WebSocketSession:
io.fabric8.kubernetes.client.KubernetesClientException: Forbidden

        at io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener.onFailure(ExecWebSocketListener.java:237)
        at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
        at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
20/07/20 16:35:24.269 [qtp1236303587-532] INFO i.l.a.w.e.ContainerExecEndpoint: Closing session..

[roiravhon]

Hi @s-bethi! looks like you get 403 from your kubernetes cluster. Is the service account Apollo uses have full permissions?

[s-bethi]

I have configured cluster admin access to apollo service account
I am still not able to access the login shell of the pod.
Not sure if i am missing anything else.

admin@loqa19:~$ kubectl describe serviceaccounts apollo
Name:                apollo
Namespace:           default
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   apollo-token-798wp
Tokens:              apollo-token-798wp
Events:              <none>

admin@loqa19:~$ kubectl describe rolebindings apollo-view
Name:         apollo-view
Labels:       <none>
Annotations:  <none>
**Role:
  Kind:  ClusterRole
  Name:  cluster-admin**
Subjects:
  Kind            Name    Namespace
  ----            ----    ---------
  **ServiceAccount  apollo  default**
admin@loqa19:~$

Here is the tcpdump of the request from apollo to kubeapi server

0:08:15.322846 IP (tos 0x0, ttl 64, id 9765, offset 0, flags [DF], proto TCP (6), length 501)
    192.168.2.3.37720 > 172.19.0.19.8080: Flags [P.], cksum 0x70b9 (incorrect -> 0x5120), seq 1:450, ack 1, win 123, options [nop,nop,TS val 215705969 ecr 215692426], length 449: HTTP, length: 449
        GET /api/v1/namespaces/default/pods/sample-apollo-app-654f5755d8-rhfc2/exec?command=%2Fbin%2Fbash&container=sample-apollo-app&tty=true&stdin=true&stdout=true&stderr=true HTTP/1.1
        Sec-WebSocket-Protocol: v4.channel.k8s.io
        Upgrade: websocket
        Connection: Upgrade
        Sec-WebSocket-Key: +ZRY+yMPW/KSj5Xwf1jmgw==
        Sec-WebSocket-Version: 13
        Authorization: Bearer $SECRET_TOKEN
        Host: 172.19.0.19:8080
        Accept-Encoding: gzip
        User-Agent: okhttp/3.12.0

00:08:15.323258 IP (tos 0x0, ttl 64, id 11768, offset 0, flags [DF], proto TCP (6), length 219)
    172.19.0.19.8080 > 192.168.2.3.37720: Flags [P.], cksum 0x6f9f (incorrect -> 0xba2b), seq 1:168, ack 450, win 126, options [nop,nop,TS val 215705969 ecr 215705969], length 167: HTTP, length: 167
        HTTP/1.1 403 Forbidden
        Content-Type: text/plain; charset=utf-8
        X-Content-Type-Options: nosniff
        Date: Wed, 22 Jul 2020 00:08:15 GMT
        Content-Length: 10

        Forbidden

[roiravhon]

Very hard to debug this, the forbidden comes from your cluster..
Are you able to run exec in kubectl to that pod?

[s-bethi]

Yes i am able to access the pod

admin@loqa19:~$ kubectl get pods | grep apollo
sample-apollo-app-654f5755d8-rhfc2   1/1     Running   0          38h
admin@loqa19:~$
admin@loqa19:~$
admin@loqa19:~$
admin@loqa19:~$ kubectl exec -ti sample-apollo-app-654f5755d8-rhfc2 -- bin/sh
/ #
/ #
/ # ls
bin    dev    etc    go.sh  home   lib    media  mnt    proc   root   run    sbin   srv    sys    tmp    usr    var
/ #
/ #
/ # date
Thu Jul 23 14:52:28 UTC 2020
/ #
/ # exit
admin@loqa19:~$ kubectl exec -ti sample-apollo-app-654f5755d8-rhfc2 -- bin/bash
OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "exec: \"bin/bash\": stat bin/bash: no such file or directory": unknown
command terminated with exit code 126
admin@loqa19:~$
admin@loqa19:~$
admin@loqa19:~$
admin@loqa19:~$

[roiravhon]

Sorry for the delayed response here -
I really have no idea what's up. Are you using RBAC? maybe you need to give explicit exec permission to the service account

[s-bethi]

Hey, sorry i couldn't figure out the access issues.
I didn't make any progress with getting to the exec shell of the pod.

[roiravhon]

The next phase is to try accessing the API with Apollo service account and see if we have anything more detailed or search kubernetes logs for any clues...
LMK if I can help