Skip to content

Commit

Permalink
Github workflow and helper scripts to build, deploy and test. (#11)
Browse files Browse the repository at this point in the history
Initial workflow to test PRs that builds the operator, deploys it and
runs a basic test on it.

Created scripts/ci folder with some helper bash scripts to build
(build.sh), install the operator in a kind cluster (deploy.sh) and run a
basic smoke test (smoke_test.sh).

The smoke test applies the sample resources in config/samples, patching
the existing kustomization.yaml to include the yamls in
config/samples/extra (configMap and secret).

Both the controller's container image and the sidecar app image are
injected by means of kustomization patches + env vars.

The sidecar image is now set using the corresponding env var
SIDECAR_APP_IMG from the controller's container manifest.
  • Loading branch information
greyerof authored Jan 30, 2024
1 parent 88bfe1f commit 36986a6
Show file tree
Hide file tree
Showing 15 changed files with 295 additions and 29 deletions.
75 changes: 74 additions & 1 deletion .github/workflows/pre-main.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -72,11 +72,84 @@ jobs:

- name: Typos
uses: crate-ci/typos@master

- name: Yamllint
uses: ibiqlik/action-yamllint@v3
with:
config_file: .yamllint.yml

- name: Go vet
run: make vet

test:
name: Build, install and test operator in a Kind cluster
runs-on: ubuntu-22.04
env:
SHELL: /bin/bash
KUBECONFIG: '/home/runner/.kube/config'

steps:
- name: Disable default go problem matcher
run: echo "::remove-matcher owner=go::"

- name: Check out code into the Go module directory
uses: actions/checkout@v4
with:
ref: ${{ github.sha }}

- name: Build operator
run: ./scripts/ci/build.sh

- name: Removed unused docker images and go cache cleanup
run: |
df -h
docker rmi $(docker images -f "dangling=true" -q) || true
docker builder prune --all -f
go clean -modcache
df -h
# Create a Kind cluster for testing.
- name: Check out `cnf-certification-test-partner` repo
uses: actions/checkout@v4
with:
repository: test-network-function/cnf-certification-test-partner
path: cnf-certification-test-partner

- name: Bootstrap cluster, docker, and python
uses: nick-fields/retry@v2
with:
timeout_minutes: 5
max_attempts: 3
command: cd ${GITHUB_WORKSPACE}/cnf-certification-test-partner; make bootstrap-cluster && make bootstrap-docker-ubuntu-local && make bootstrap-python-ubuntu-local

- name: Run 'make rebuild-cluster'
uses: nick-fields/retry@v2
env:
SKIP_PRELOAD_IMAGES: true
with:
timeout_minutes: 15
max_attempts: 3
command: cd ${GITHUB_WORKSPACE}/cnf-certification-test-partner; make rebuild-cluster

- name: Run 'make install'
uses: nick-fields/retry@v2
env:
SKIP_PRELOAD_IMAGES: true
with:
timeout_minutes: 20
max_attempts: 3
command: cd ${GITHUB_WORKSPACE}/cnf-certification-test-partner; make install

- name: More cleanup
run: |
df -h
docker rmi $(docker images -f "dangling=true" -q) || true
docker builder prune --all -f
go clean -modcache
df -h
- name: Install operator in the Kind cluster
run: ./scripts/ci/deploy.sh

- name: Run simple smoke test.
run: ./scripts/ci/smoke_test.sh
14 changes: 8 additions & 6 deletions bundle/manifests/tnf-op.clusterserviceversion.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ metadata:
"app.kubernetes.io/part-of": "tnf-op"
},
"name": "cnfcertificationsuiterun-sample",
"namespace": "cnf-certification-operator"
"namespace": "cnf-certsuite-operator"
},
"spec": {
"configMapName": "cnf-certsuite-config",
Expand All @@ -28,10 +28,10 @@ metadata:
}
]
capabilities: Basic Install
createdAt: "2024-01-12T12:33:33Z"
createdAt: "2024-01-25T07:40:11Z"
operators.operatorframework.io/builder: operator-sdk-v1.33.0
operators.operatorframework.io/project_layout: go.kubebuilder.io/v3
name: tnf-op.v0.0.3
name: tnf-op.v0.0.1
namespace: placeholder
spec:
apiservicedefinitions: {}
Expand Down Expand Up @@ -157,8 +157,10 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.annotations['olm.targetNamespaces']
image: quay.io/greyerof/cnf-op:initialtestv5
imagePullPolicy: Always
- name: SIDECAR_APP_IMG
value: quay.io/testnetworkfunction/cnf-certsuite-operator-sidecar:v0.0.1
image: quay.io/testnetworkfunction/cnf-certsuite-operator:v0.0.1
imagePullPolicy: IfNotPresent
livenessProbe:
httpGet:
path: /healthz
Expand Down Expand Up @@ -307,4 +309,4 @@ spec:
maturity: alpha
provider:
name: RedHat
version: 0.0.3
version: 0.0.1
2 changes: 1 addition & 1 deletion cnf-cert-sidecar/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM registry.access.redhat.com/ubi8/ubi:8.9-1028 AS build
FROM registry.access.redhat.com/ubi8/ubi:8.9-1107.1705420509 AS build

# Install dependencies
# hadolint ignore=DL3041
Expand Down
2 changes: 1 addition & 1 deletion config/default/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Adds namespace to all resources.
namespace: cnf-certification-operator
namespace: cnf-certsuite-operator

# Value of this field is prepended to the
# names of all resources, e.g. a deployment named
Expand Down
5 changes: 3 additions & 2 deletions config/manager/kustomization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,6 @@ apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
images:
- name: controller
newName: quay.io/greyerof/cnf-op
newTag: initialtestv5
newName: quay.io/testnetworkfunction/cnf-certsuite-operator
newTag: v0.0.1

4 changes: 3 additions & 1 deletion config/manager/manager.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -75,9 +75,11 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: SIDECAR_APP_IMG
value: quay.io/testnetworkfunction/cnf-certsuite-operator-sidecar:v0.0.1
image: controller:latest
name: manager
imagePullPolicy: Always
imagePullPolicy: IfNotPresent
securityContext:
allowPrivilegeEscalation: false
capabilities:
Expand Down
2 changes: 1 addition & 1 deletion config/samples/kustomization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,5 @@ resources:

#+kubebuilder:scaffold:manifestskustomizesamples

namespace: cnf-certification-operator
namespace: cnf-certsuite-operator

8 changes: 5 additions & 3 deletions controllers/cnf-cert-job/cnfcertjob.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,10 @@ type Config struct {
LogLevel string
ConfigMapName string
PreflightSecretName string
SideCarAppImage string
}

func NewConfig(podName, namespace, certSuiteConfigRunName, labelsFilter, logLevel, configMapName, preflightSecretName string) *Config {
func NewConfig(podName, namespace, certSuiteConfigRunName, labelsFilter, logLevel, configMapName, preflightSecretName, sideCarAppImage string) *Config {
return &Config{
PodName: podName,
Namespace: namespace,
Expand All @@ -33,6 +34,7 @@ func NewConfig(podName, namespace, certSuiteConfigRunName, labelsFilter, logLeve
LogLevel: logLevel,
ConfigMapName: configMapName,
PreflightSecretName: preflightSecretName,
SideCarAppImage: sideCarAppImage,
}
}

Expand All @@ -50,7 +52,7 @@ func New(config *Config) *corev1.Pod {
Containers: []corev1.Container{
{
Name: definitions.CnfCertSuiteSidecarContainerName,
Image: "quay.io/greyerof/cnf-op:sidecarv4",
Image: config.SideCarAppImage,
Env: []corev1.EnvVar{
{
Name: "MY_POD_NAME",
Expand All @@ -77,7 +79,7 @@ func New(config *Config) *corev1.Pod {
Value: config.CertSuiteConfigRunName,
},
},
ImagePullPolicy: "Always",
ImagePullPolicy: "IfNotPresent",
VolumeMounts: []corev1.VolumeMount{
{
Name: "cnf-certsuite-output",
Expand Down
12 changes: 11 additions & 1 deletion controllers/cnfcertificationsuiterun_controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ package controllers
import (
"context"
"fmt"
"os"
"time"

corev1 "k8s.io/api/core/v1"
Expand All @@ -39,6 +40,8 @@ import (
"github.com/sirupsen/logrus"
)

var sideCarImage string

// CnfCertificationSuiteRunReconciler reconciles a CnfCertificationSuiteRun object
type CnfCertificationSuiteRunReconciler struct {
client.Client
Expand Down Expand Up @@ -239,7 +242,8 @@ func (r *CnfCertificationSuiteRunReconciler) Reconcile(ctx context.Context, req
cnfrun.Spec.LabelsFilter,
cnfrun.Spec.LogLevel,
cnfrun.Spec.ConfigMapName,
cnfrun.Spec.PreflightSecretName)
cnfrun.Spec.PreflightSecretName,
sideCarImage)
cnfCertJobPod := cnfcertjob.New(config)

err := r.Create(ctx, cnfCertJobPod)
Expand All @@ -260,6 +264,12 @@ func (r *CnfCertificationSuiteRunReconciler) SetupWithManager(mgr ctrl.Manager)
logrus.Infof("Setting up CnfCertificationSuiteRunReconciler's manager.")
certificationRuns = map[certificationRun]string{}

var found bool
sideCarImage, found = os.LookupEnv(definitions.SideCarImageEnvVar)
if !found {
return fmt.Errorf("sidecar app img env var %q not found", definitions.SideCarImageEnvVar)
}

return ctrl.NewControllerManagedBy(mgr).
For(&cnfcertificationsv1alpha1.CnfCertificationSuiteRun{}).
WithEventFilter(ignoreUpdatePredicate()).
Expand Down
2 changes: 2 additions & 0 deletions controllers/definitions/definitions.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,6 @@ const (
PreflightDockerConfigFilePath = CnfPreflightConfigFolder + "/preflight_dockerconfig.json"

SideCarResultsFolderEnvVar = "TNF_RESULTS_FOLDER"

SideCarImageEnvVar = "SIDECAR_APP_IMG"
)
12 changes: 0 additions & 12 deletions hack/deleteall.sh

This file was deleted.

46 changes: 46 additions & 0 deletions scripts/ci/build.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
#!/bin/bash
#
# Builds the operator's controller and sidecar images. The manager's env var
# with the sidecar app image is also replaced with the new name, which is set
# using the value from env var SIDECAR_IMG. It will be defaulted to DEFAULT_SIDECAR_IMG
# if that env var is not found.
#
# This script is intended to be used in CI workflows to create loca/test images just for
# testing purposes, but can also be used to generate images for new official releases. In
# that case, the following env vars should have been exported before running this script:
# VERSION : The desired new/test version tag that will be applied to both images.
# IMG : The image for the controller's container.
#

# Bash settings: display (expanded) commands and fast exit on first error.
set -o xtrace
set -o errexit

DEFAULT_TEST_VERSION="0.0.1-test"
DEFAULT_SIDECAR_IMG="local-test-sidecar-image:v$DEFAULT_TEST_VERSION"
DEFAULT_IMG="ci-cnf-op:v$DEFAULT_TEST_VERSION"

export VERSION="${VERSION:-$DEFAULT_TEST_VERSION}"
export SIDECAR_IMG="${SIDECAR_IMG:-$DEFAULT_SIDECAR_IMG}"
export IMG="${IMG:-$DEFAULT_IMG}"

# step: Build manifests and controller app
make build

# step: Run tests
make test

# step: Build sidecar app
docker build --no-cache -t "${SIDECAR_IMG}" -f cnf-cert-sidecar/Dockerfile .

# Local install kustomize app that is needed to edit/patch the kustomization.yaml
make kustomize

# step: Add env var to the controller's container to set the sidecar image app that was built right before
pushd config/manager
../../bin/kustomize edit add patch --kind Deployment --patch "[{\"op\": \"replace\", \"path\": \"/spec/template/spec/containers/0/env/1\", \"value\": {\"name\": \"SIDECAR_APP_IMG\", \"value\": \"${SIDECAR_IMG}\"} }]"
popd

# step: Build docker image for the controller. This will use IMG pointing to local docker image for the controller, which
# will be pushed to kind with "docker load docker-image $IMG"
make docker-build
41 changes: 41 additions & 0 deletions scripts/ci/deploy.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
#!/bin/bash
#
# This script deploys a recently built operator in a kind cluster.
# Both the operator's controller and the sidecar app images are
# preloaded into the kind cluster's nodes to avoid the need of
# uploading the images to an external registry (quay/docker).
#
# The operator is deployed in the namespace set by env var CNF_CERTSUITE_OPERATOR_NAMESPACE
# or in the defaulted namespace "cnf-certsuite-operator" if that env var is not found.
#

# Bash settings: display (expanded) commands and fast exit on first error.
set -o xtrace
set -o errexit

DEFAULT_CNF_CERTSUITE_OPERATOR_NAMESPACE="cnf-certsuite-operator"
DEFAULT_TEST_VERSION="0.0.1-test"
DEFAULT_SIDECAR_IMG="local-test-sidecar-image:v$DEFAULT_TEST_VERSION"
DEFAULT_IMG="ci-cnf-op:v$DEFAULT_TEST_VERSION"

CNF_CERTSUITE_OPERATOR_NAMESPACE=${CNF_CERTSUITE_OPERATOR_NAMESPACE:-$DEFAULT_CNF_CERTSUITE_OPERATOR_NAMESPACE}

export VERSION="${VERSION:-$DEFAULT_TEST_VERSION}"
export SIDECAR_IMG="${SIDECAR_IMG:-$DEFAULT_SIDECAR_IMG}"
export IMG="${IMG:-$DEFAULT_IMG}"

kind load docker-image "${SIDECAR_IMG}"
kind load docker-image "${IMG}"

# "make deploy" uses the IMG env var internally, and it needs to be exported.
# let's patch the installation namespace.
make kustomize

pushd config/default
../../bin/kustomize edit set namespace "${CNF_CERTSUITE_OPERATOR_NAMESPACE}"
popd

make deploy

# step: Wait for the controller's containers to be ready
oc wait --for=condition=ready pod --all=true -n cnf-certsuite-operator --timeout=2m
41 changes: 41 additions & 0 deletions scripts/ci/resource_exists.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
#!/bin/bash
#
# WARNING: This is a helper script to be run with the "timeout" command, like:
# $ timeout 60s $0 pod mypodname mynamespace
#
# Polls the cluster with "oc get ..." every N secs and returns with 0 status
# code as soon as the given resource by name is found in a given namespace.
# In case the resource is not found or there's any problem with "oc get", the
# script will never return, as it's polling every N secs, no matter the type of
# error was returned.
#
# $1 resource kind, e.g. pod, deployment...
# $2 name
# $3 namespace
# $4 check interval time (optional, defaults to 5 segs)
#
# Examples:
# $0 pod mypodname mypodnamespace

DEFAULT_INTERVAL_CHECK_SEGS=5
INTERVAL_CHECK_SEGS=${DEFAULT_INTERVAL_CHECK_SEGS}

RESOURCE_KIND=$1
RESOURCE_NAME=$2
NAMESPACE=$3

if [ "$4" != "" ] ; then
INTERVAL_CHECK_SEGS=$4
fi

echo "Polling every ${INTERVAL_CHECK_SEGS} secs for ${RESOURCE_NAME} (kind: ${RESOURCE_KIND}) to exist in namespace ${NAMESPACE}"

while true; do
if oc get "${RESOURCE_KIND}" -n "${NAMESPACE}" "${RESOURCE_NAME}" ; then
echo "${RESOURCE_NAME} (kind: ${RESOURCE_KIND}) found in namespace ${NAMESPACE}."
exit 0
fi

echo "${RESOURCE_NAME} (kind: ${RESOURCE_KIND}) not found yet in namespace ${NAMESPACE}..."
sleep "${INTERVAL_CHECK_SEGS}"
done
Loading

0 comments on commit 36986a6

Please sign in to comment.