Bring up Kubernetes Cluster for Eiffel Community

[fdegir]

Description

There is an idea to bring up a Kubernetes Cluster for Eiffel Community in order to use it as part of Eiffel CI/CD to deploy Eiffel components for testing purposes. In addition to using it for CI/CD this could be a step toward bringing a reference Eiffel deployment as well.

Motivation

Eiffel Community does not have extensive testing at the moment.
This will allow community to explore ways to extend the test scope for Eiffel Components as well as open up further opportunities to deploy Eiffel for real in open source.

Exemplification

Many communities have such environments to run tests or use for production purposes.

Benefits

This will allow community to explore ways to extend the test scope for Eiffel Components as well as open up further opportunities to deploy Eiffel for real in open source.

Possible Drawbacks

This will require Eiffel Community members to put some time on maintaining the Kubernetes Cluster and adjust Eiffel CI/CD for deploying Eiffel components on it.

[fdegir]

@t-persson this is the action point from the last TC meeting. Could you please pass the details of Kubernetes Cluster you have in mind like k8s version, networking details, storage details, no of worker nodes, specs of worker nodes and anything else you could think of?

[t-persson]

Yes, I am working on it :)

[t-persson]

K8S version: We are running on 1.17.3 now but we would want to run on latest.

Networking:

• External access to ETOS API with a hostname.
• Internal access to all other services.
• ETOS Test Runner must be able to access ETOS API from an external source (if running in Jenkins).

Storage details:

• ETOS requires an event repository and as such a MongoDB with storage capacity for all Eiffel events is required.
• 8GB Volumes per redis instance (see below)

Worker nodes:
I know very little about hardware requirements
This is what we are running today for a single deployment:

• ETOS
  • 1 Pod ETOS API
  • 1 Pod ETOS Suite Starter
  • 2-10 Pods ETOS Environment Provider
  • 1 Job ETOS Suite Runner per test suite execution
  • 1 Job ETOS Test Runner per sub suite (at least 1, only if k8s is main execution space)
  • 4 Pods Redis
  • 5 Pods EiffelGraphql API
• Infrastructure
  • 3 Pods MongoDB
  • 5 Pods EiffelGraphql Storage
  • And we would also require a RabbitMQ service somewhere

This is our current specs (might be too much for Eiffel community):

	Cores	Memory	Available pods
Worker 1	16	31.3GiB	110
Worker 2	16	31.3GiB	110
Worker 3	16	31.3GiB	110

Service accounts
We need to be able to start Kubernetes jobs from a service account within the ETOS Suite Starter (and within the ETOS Suite Runner, if Kubernetes is the main execution space).
It would also be great to have access to delete Jobs from a service account, so we can create CronJobs that delete old jobs.

Authentication
Since this would run on the Internet instead of our internal network we would need to handle authentication on the ETOS API. This is not something that we have today and it would be awesome if there was a service we could piggyback on so that the Eiffel community gets access.

@fredjn @andjoe-axis Please add more if I'm missing something.
@fdegir Please ask me if I've not answered your question :)

[fdegir]

Thanks for this @t-persson.
I've been thinking about how to approach this since morning and will come back to you with a proposal or two.

[fdegir]

Notes from the meeting between Eiffel and Nordix Communities on 2021-09-15.

Participants:

• @t-persson
• @m-linner-ericsson
• @toomer
• @fdegir
• @e-backmark-ericsson (partly)

Notes:

• The aim is to have this hosted in Nordix Community. This needs to be blessed by Nordix Board and TC.
• Eiffel Community project ETOS is launched on Kubernetes together with the infrastructure required for it so ETOS development could be done in open source.
• This could also enable publishing Eiffel Messages from Eiffel Community so the contributing companies could hook into Eiffel deployment of Eiffel Community.
• This specs for the cluster is based on what some community members are currently using. This could be a starting point and if there are issues with resourcing, it could be slimmed down. At the start, there will be single master and then we see if this is sufficient going forward.
• If Nordix Board approves hosting Kubernetes on Nordix Infra, a separate project for Eiffel could be created on Nordix Tenant and the contributors nominated by Eiffel TC could be granted rights to operate and maintain the cluster.
• At the beginning, the proposal is to use Nordix Cloud Infra Automation Framework (aka engine) to deploy Kubernetes. If Eiffel Community thinks this installation is not sufficient, the community could install Kubernetes their own way. However this requires some exposure to OpenStack, especially to openstackcli.
• The Kubernetes version to use is v1.20.x based on what is supported by engine.
• It may be necessary for Kubernetes cluster to be accessible publicly. What this means exactly remains to be seen based on the requirements from Eiffel Community. But the main principle is to have Kubernetes deployment as secure as possible, possibly behind VPN.
• Background is available here: https://hackmd.io/X3Nrxe4MRdSJI_lsTkSwiQ

APs:

• @t-persson and @m-linner-ericsson to reach out to Eiffel TC to nominate 2 Eiffel Community members to collaborate with Nordix Community closely.
• @fdegir and @toomer to reach out to Nordix Board for blessing.

[e-backmark-ericsson]

Additional comments in old TC MoM: https://github.com/eiffel-community/community/blob/master/meetings/MEETINGS_TC_2023.md#january-11-2023 (see Action Items)

[e-backmark-ericsson]

An Easy2Use deployment on Nordix was presented on the Eiffel Summit in Gothenburg last week

[e-backmark-ericsson]

@pef-ericsson to look into https://hackmd.io/X3Nrxe4MRdSJI_lsTkSwiQ?view and consider what the relevant next steps would be

[pef-ericsson]

@e-backmark-ericsson @m-linner-ericsson @t-persson

• A self-hosted runner has been configured with the Eiffel Nordix-dedicated cluster with the support of Nordix organization.
• The runner has been tested by Nordix and it seems to working fine now. It can reach other servers in the private network as well. I will continue implementing some GitHub workflows for Easy2Use project in the near future.
• At the moment, only projects/repos under Nordix can be configured with the corresponding runner. There are some desired repos according to hackmd doc which should be added later.
• An example of a GitHub Action workflow. See that the runs-on is defined as eiffel-runner-set.

name: CI 
   on:
  push:
    branches: [ master ]
jobs:
  build:
    runs-on: eiffel-runner-set
    steps:
      - uses: actions/checkout@v3

[m-linner-ericsson]

Continuation of this work happen under #207