Skip to content

Latest commit

 

History

History
195 lines (134 loc) · 8.15 KB

getting_started_locally.md

File metadata and controls

195 lines (134 loc) · 8.15 KB

Running Gardener locally

This document will walk you through running Gardener on your local machine for development purposes. If you encounter difficulties, please open an issue so that we can make this process easier.

Gardener runs in any Kubernetes cluster. In this guide, we will start a KinD cluster which is used as both garden and seed cluster (please refer to the architecture overview) for simplicity.

The Gardener components, however, will be run as regular processes on your machine (hence, no container images are being built).

Architecture Diagram

Prerequisites

  • Make sure that you have followed the Local Setup guide up until the Get the sources step.

  • Make sure your Docker daemon is up-to-date, up and running and has enough resources (at least 4 CPUs and 4Gi memory; see here how to configure the resources for Docker for Mac).

    Please note that 4 CPU / 4Gi memory might not be enough for more than one Shoot cluster, i.e., you might need to increase these values if you want to run additional Shoots. If you plan on following the optional steps to create a second seed cluster, the required resources will be more - at least 10 CPUs and 16Gi memory.

    Additionally, please configure at least 120Gi of disk size for the Docker daemon.

    Tip: With docker system df and docker system prune -a you can cleanup unused data.

  • Make sure that you increase the maximum number of open files on your host:

    • On Mac, run sudo launchctl limit maxfiles 65536 200000

    • On Linux, extend the /etc/security/limits.conf file with

      * hard nofile 97816
* soft nofile 97816

      and reload the terminal.

Setting up the KinD cluster (garden and seed)

make kind-up KIND_ENV=local

This command sets up a new KinD cluster named gardener-local and stores the kubeconfig in the ./example/gardener-local/kind/kubeconfig file.

It might be helpful to copy this file to $HOME/.kube/config since you will need to target this KinD cluster multiple times. Alternatively, make sure to set your KUBECONFIG environment variable to ./example/gardener-local/kind/kubeconfig for all future steps via export KUBECONFIG=example/gardener-local/kind/kubeconfig.

All following steps assume that your are using this kubeconfig.

Setting up Gardener

make dev-setup                                                                # preparing the environment (without webhooks for now)
kubectl wait --for=condition=ready pod -l run=etcd -n garden --timeout 2m     # wait for etcd to be ready
make start-apiserver                                                          # starting gardener-apiserver

In a new terminal pane, run

kubectl wait --for=condition=available apiservice v1beta1.core.gardener.cloud # wait for gardener-apiserver to be ready
make start-admission-controller                                               # starting gardener-admission-controller

In a new terminal pane, run

make dev-setup DEV_SETUP_WITH_WEBHOOKS=true                                   # preparing the environment with webhooks
make start-controller-manager                                                 # starting gardener-controller-manager

(Optional): In a new terminal pane, run

make start-scheduler                                                          # starting gardener-scheduler

In a new terminal pane, run

make register-local-env                                                       # registering the local environment (CloudProfile, Seed, etc.)
make start-gardenlet SEED_NAME=local                                          # starting gardenlet

In a new terminal pane, run

make start-extension-provider-local                                           # starting gardener-extension-provider-local

ℹ️ The provider-local is started with elevated privileges since it needs to manipulate your /etc/hosts file to enable you accessing the created shoot clusters from your local machine, see this for more details.

Creating a Shoot cluster

You can wait for the Seed to be ready by running

kubectl wait --for=condition=gardenletready --for=condition=extensionsready --for=condition=bootstrapped seed local --timeout=5m

Alternatively, you can run kubectl get seed local and wait for the STATUS to indicate readiness:

NAME    STATUS   PROVIDER   REGION   AGE     VERSION       K8S VERSION
local   Ready    local      local    4m42s   vX.Y.Z-dev    v1.21.1

In order to create a first shoot cluster, just run

kubectl apply -f example/provider-local/shoot.yaml

You can wait for the Shoot to be ready by running

kubectl wait --for=condition=apiserveravailable --for=condition=controlplanehealthy --for=condition=everynodeready --for=condition=systemcomponentshealthy shoot local -n garden-local --timeout=10m

Alternatively, you can run kubectl -n garden-local get shoot local and wait for the LAST OPERATION to reach 100%:

NAME    CLOUDPROFILE   PROVIDER   REGION   K8S VERSION   HIBERNATION   LAST OPERATION            STATUS    AGE
local   local          local      local    1.21.0        Awake         Create Processing (43%)   healthy   94s

(Optional): You could also execute a simple e2e test (creating and deleting a shoot) by running

make test-e2e-local-simple KUBECONFIG="$PWD/example/gardener-local/kind/kubeconfig"

When the shoot got successfully created you can access it as follows:

kubectl -n garden-local get secret local.kubeconfig -o jsonpath={.data.kubeconfig} | base64 -d > /tmp/kubeconfig-shoot-local.yaml
kubectl --kubeconfig=/tmp/kubeconfig-shoot-local.yaml get nodes

(Optional): Setting up a second seed cluster

There are cases where you would want to create a second seed cluster in your local setup. For example, if you want to test the control plane migration feature. The following steps describe how to do that.

Add a new IP address on your loopback device which will be necessary for the new KinD cluster that you will create. On Mac, the default loopback device is lo0.

sudo ip addr add 127.0.0.2 dev lo0                                     # adding 127.0.0.2 ip to the loopback interface

Next, setup the second KinD cluster:

make kind2-up KIND_ENV=local

This command sets up a new KinD cluster named gardener-local2 and stores its kubeconfig in the ./example/gardener-local/kind2/kubeconfig file. You will need this file when starting the provider-local extension controller for the second seed cluster.

make register-kind2-env                                           # registering the local2 seed
make start-gardenlet SEED_NAME=local2                             # starting gardenlet for the local2 seed

In a new terminal pane, run

export KUBECONFIG=./example/gardener-local/kind2/kubeconfig       # setting KUBECONFIG to point to second kind cluster
make start-extension-provider-local \
  WEBHOOK_SERVER_PORT=9444 \
  WEBHOOK_CERT_DIR=/tmp/gardener-extension-provider-local2 \
  SERVICE_HOST_IP=127.0.0.2 \
  METRICS_BIND_ADDRESS=:8082 \
  HEALTH_BIND_ADDRESS=:8083                                       # starting gardener-extension-provider-local

If you want to perform a control plane migration you can follow the steps outlined here to migrate the shoot cluster to the second seed you just created.

Deleting the Shoot cluster

./hack/usage/delete shoot local garden-local

(Optional): Tear down the second seed cluster

make tear-down-kind2-env
make kind2-down

Tear down the Gardener environment

make tear-down-local-env
make kind-down

Further reading

This setup makes use of the local provider extension. You can read more about it in this document.