Skip to content

Latest commit

 

History

History
359 lines (301 loc) · 15.4 KB

CLUSTER.md

File metadata and controls

359 lines (301 loc) · 15.4 KB

Notes on how to structure project for multi k8s cluster

Setting up a GitHub repository for developing on multiple Kubernetes clusters requires a clear and organized folder structure. The goal is to keep your environment configurations, manifests, and scripts separated by cluster, while also making it easy to manage deployments across different clusters. Here's an example of how you could organize the folder structure:

Example Folder Structure for Multi-Cluster Setup

akello-repo/
│
├── clusters/
│   ├── core-services-cluster/
│   │   ├── manifests/
│   │   │   ├── deployment.yaml
│   │   │   ├── service.yaml
│   │   │   └── configmap.yaml
│   │   ├── helm/
│   │   │   └── charts/
│   │   │       ├── core-service-chart/
│   │   │       └── values.yaml
│   │   ├── scripts/
│   │   │   ├── create_cluster.sh
│   │   │   └── deploy_core_services.sh
│   │   └── README.md
│   │
│   ├── ai-ml-cluster/
│   │   ├── manifests/
│   │   │   ├── deployment.yaml
│   │   │   └── model-inference-service.yaml
│   │   ├── helm/
│   │   │   └── charts/
│   │   │       └── ai-ml-service-chart/
│   │   ├── scripts/
│   │   │   ├── create_cluster.sh
│   │   │   └── deploy_ai_ml_services.sh
│   │   └── README.md
│   │
│   ├── iot-cluster/
│   │   ├── manifests/
│   │   │   ├── mqtt-service.yaml
│   │   │   └── sensor-data-ingestion.yaml
│   │   ├── helm/
│   │   │   └── charts/
│   │   │       └── iot-service-chart/
│   │   ├── scripts/
│   │   │   ├── create_cluster.sh
│   │   │   └── deploy_iot_services.sh
│   │   └── README.md
│   │
│   └── common/
│       ├── manifests/
│       │   └── common-resources.yaml
│       └── helm/
│           └── charts/
│               └── common-resources-chart/
│
├── deployment-scripts/
│   ├── setup_all_clusters.sh   # Creates all clusters
│   ├── deploy_all.sh           # Deploys all services across clusters
│   ├── destroy_all_clusters.sh # Deletes all clusters
│   └── README.md
│
├── docs/
│   ├── architecture-diagram.png
│   ├── setup-guide.md
│   └── cluster-overview.md
│
├── ci-cd/
│   ├── .github/
│   │   └── workflows/
│   │       ├── deploy-core-services.yml
│   │       ├── deploy-ai-ml-services.yml
│   │       └── deploy-iot-services.yml
│   ├── Jenkinsfile   # If you're using Jenkins
│   └── gitlab-ci.yml # If you're using GitLab CI
│
└── README.md

Folder Breakdown

1. clusters/

This folder contains subfolders for each Kubernetes cluster. Each subfolder has its own:

  • Manifests: Kubernetes YAML files specific to that cluster.

  • Helm Charts: If you’re using Helm for templating and managing deployments.

  • Scripts: Shell scripts for setting up the cluster and deploying services.

  • core-services-cluster/:

    • This folder contains manifests and Helm charts related to core services like identity management, EHR, or other business logic.
    • It also has a scripts/ folder to automate cluster creation and service deployment.

  • ai-ml-cluster/:

    • Contains the manifests and Helm charts for your AI/ML workloads, such as model training and inference services.
    • The scripts/ folder includes deployment scripts specific to AI-related services.

  • iot-cluster/:

    • Manifests and Helm charts for handling IoT data ingestion, MQTT brokers, and sensor data services.
    • This folder will have similar deployment scripts for the IoT services.

  • common/:

    • This is where you store any common resources that are shared across multiple clusters, such as shared secrets, namespaces, or configurations.
    • Both manifests and Helm charts are provided here for common services.

2. deployment-scripts/

This folder includes high-level scripts that can:

  • Create all clusters using Kind, Minikube, or your preferred local Kubernetes setup.
  • Deploy all services across the clusters, making use of the scripts in the individual cluster directories.
  • Tear down clusters when needed.

These scripts help to automate the entire multi-cluster setup locally and are useful for spinning up or tearing down environments quickly.

3. docs/

  • This is where you keep documentation for how to set up and maintain the clusters, architecture diagrams, and information about the purpose of each cluster.
  • This folder will guide team members through the reasoning behind the structure, the cluster roles, and how to work within the environment.

4. ci-cd/

  • This contains your Continuous Integration and Continuous Deployment (CI/CD) configurations, like GitHub Actions workflows, Jenkins pipelines, or GitLab CI configurations.
  • Each CI workflow file (e.g., .github/workflows/) is tailored for deploying specific clusters/services automatically when code is pushed to the relevant branches.

5. README.md

  • Each folder should have a README.md that explains what’s inside the folder, its purpose, and instructions on how to use the scripts or resources within it.

Workflow

  1. Cluster-Specific Development:

    • When you develop for a specific cluster, you work inside its respective folder (core-services-cluster, ai-ml-cluster, etc.).
    • Manifests, Helm charts, and deployment scripts for that cluster are stored here, allowing you to manage them independently from other clusters.

  2. Deploying and Managing Services:

    • You can deploy the clusters and services by running the deployment scripts in deployment-scripts/. For example, you might run setup_all_clusters.sh to spin up all your clusters, followed by deploy_all.sh to deploy all services across clusters.

  3. CI/CD Integration:

    • Your CI/CD pipelines in ci-cd/ can be set up to automatically deploy changes to specific clusters based on which parts of the repository are updated. For example, a change in the ai-ml-cluster folder could trigger a workflow that only deploys the AI/ML services to the ai-ml-cluster.

Benefits of This Structure

  • Modularity: Each cluster's configuration is separated, making it easy to develop, test, and deploy independently.
  • Automation: With the deployment scripts, you can quickly spin up, tear down, and update your entire multi-cluster setup locally.
  • Scalability: As your clusters grow, this structure can be extended to accommodate additional clusters and services without clutter.
  • CI/CD Ready: The structure is ready to integrate into your CI/CD pipeline, ensuring a smooth development process and deployment.

By organizing your repository this way, you can manage multiple clusters locally with ease and maintain consistency across your development workflow.

Notes with Terraform

If you want to use Terraform within this structured approach to manage your multiple Kubernetes clusters and their resources, you can further enhance your repository by organizing Terraform code in a way that supports modularity, reusability, and scalability. Here’s how you can integrate Terraform into the folder structure for managing clusters and resources.

Example Folder Structure with Terraform

akello-repo/
│
├── clusters/
│   ├── core-services-cluster/
│   │   ├── manifests/
│   │   ├── helm/
│   │   ├── scripts/
│   │   ├── terraform/                # New folder for Terraform code
│   │   │   ├── main.tf
│   │   │   ├── variables.tf
│   │   │   ├── outputs.tf
│   │   │   ├── providers.tf
│   │   │   └── README.md
│   │   └── README.md
│   │
│   ├── ai-ml-cluster/
│   │   ├── manifests/
│   │   ├── helm/
│   │   ├── scripts/
│   │   ├── terraform/
│   │   │   ├── main.tf
│   │   │   ├── variables.tf
│   │   │   ├── outputs.tf
│   │   │   ├── providers.tf
│   │   │   └── README.md
│   │   └── README.md
│   │
│   ├── iot-cluster/
│   │   ├── manifests/
│   │   ├── helm/
│   │   ├── scripts/
│   │   ├── terraform/
│   │   │   ├── main.tf
│   │   │   ├── variables.tf
│   │   │   ├── outputs.tf
│   │   │   ├── providers.tf
│   │   │   └── README.md
│   │   └── README.md
│   │
│   └── common/
│       ├── terraform/
│       │   ├── main.tf               # Common resources, like VPCs, subnets, etc.
│       │   ├── variables.tf
│       │   ├── outputs.tf
│       │   ├── providers.tf
│       │   └── README.md
│       ├── manifests/
│       └── helm/
│
├── deployment-scripts/
│   ├── setup_all_clusters.sh
│   ├── deploy_all.sh
│   └── README.md
│
├── terraform-modules/                 # Centralized modules for reuse
│   ├── k8s-cluster/                   # Reusable module for Kubernetes cluster creation
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── outputs.tf
│   ├── k8s-services/                  # Module for service deployment (e.g., Helm or YAML)
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── outputs.tf
│   └── networking/                    # Module for setting up networking (VPC, subnets)
│       ├── main.tf
│       ├── variables.tf
│       └── outputs.tf
│
├── docs/
│   └── architecture-diagram.png
│
└── README.md

Key Components

1. Terraform in Each Cluster Folder

Each cluster folder (e.g., core-services-cluster, ai-ml-cluster, etc.) will have a terraform/ directory where the Terraform configuration for managing that cluster is stored. This setup gives each cluster its own Terraform configuration, allowing you to manage them independently if needed.

  • main.tf: Defines the infrastructure for the specific cluster. This will include Kubernetes resources (or cloud infrastructure resources, such as managed Kubernetes clusters like EKS, GKE, or AKS) required for that specific cluster.
  • variables.tf: Contains the input variables needed to customize the Terraform configuration for the specific cluster.
  • providers.tf: Defines the providers, such as the cloud provider (AWS, GCP, Azure), or local Kubernetes provider (like kind or minikube), that Terraform will interact with.
  • outputs.tf: Defines the outputs (e.g., cluster endpoint, authentication tokens, etc.) that are needed for other parts of the setup or for using kubectl to interact with the cluster.

2. Centralized Terraform Modules (terraform-modules/)

  • The terraform-modules/ directory holds reusable modules, which are self-contained Terraform configurations that can be used across multiple clusters. These modules help reduce duplication and make your setup more modular and maintainable.

Some potential modules:

  • k8s-cluster/: A module that defines how to create a Kubernetes cluster, either locally (with tools like kind or minikube) or on the cloud (using EKS, GKE, or AKS).
  • k8s-services/: A module that defines how to deploy Kubernetes services, either using raw Kubernetes YAML manifests or Helm charts.
  • networking/: A module for setting up networking infrastructure (like VPCs, subnets, and load balancers), shared across clusters.

These modules can be referenced in each cluster's main.tf file to reuse the logic for cluster creation, service deployment, or networking setup.

3. Common Resources (common/terraform/)

  • The common/terraform/ folder contains Terraform configurations for resources that are shared across multiple clusters, such as VPCs, subnets, DNS, or other networking components. This helps keep shared infrastructure components separated from cluster-specific resources, allowing easier management and reuse.

4. Deployment Scripts (deployment-scripts/)

You can add Terraform-specific commands to the existing deployment scripts:

  • setup_all_clusters.sh: This script can orchestrate the setup of all clusters by running Terraform commands for each cluster. 
    #!/bin/bash
# Initialize and apply Terraform for core services cluster
cd ../clusters/core-services-cluster/terraform/
terraform init
terraform apply -auto-approve

# Initialize and apply Terraform for AI/ML cluster
cd ../../ai-ml-cluster/terraform/
terraform init
terraform apply -auto-approve

# Initialize and apply Terraform for IoT cluster
cd ../../iot-cluster/terraform/
terraform init
terraform apply -auto-approve
  • destroy_all_clusters.sh: This script can destroy all the clusters.
  • deploy_all.sh: This can call both the Terraform apply and Kubernetes/Helm deployment scripts.

Example of Terraform Code

Example of main.tf for core-services-cluster

provider "aws" {
  region = var.aws_region
}

module "networking" {
  source  = "../../terraform-modules/networking"
  vpc_cidr = var.vpc_cidr
}

module "k8s_cluster" {
  source = "../../terraform-modules/k8s-cluster"
  cluster_name = var.cluster_name
  vpc_id       = module.networking.vpc_id
}

output "cluster_endpoint" {
  value = module.k8s_cluster.endpoint
}

output "cluster_token" {
  value = module.k8s_cluster.token
}

Example variables.tf

variable "aws_region" {
  description = "The AWS region to deploy the cluster"
  type        = string
  default     = "us-west-2"
}

variable "vpc_cidr" {
  description = "CIDR block for the VPC"
  type        = string
  default     = "10.0.0.0/16"
}

variable "cluster_name" {
  description = "The name of the Kubernetes cluster"
  type        = string
  default     = "core-services-cluster"
}

Using Terraform with GitHub Actions (CI/CD)

In the ci-cd/ folder, you can define Terraform workflows as part of your GitHub Actions to automatically apply Terraform changes when updates are pushed to your repo:

# .github/workflows/terraform.yml
name: "Terraform Apply"

on:
  push:
    branches:
      - main

jobs:
  terraform:
    name: "Terraform Apply"
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - name: "Set up Terraform"
      uses: hashicorp/setup-terraform@v1
    - name: "Terraform Init"
      run: terraform init
      working-directory: ./clusters/core-services-cluster/terraform/
    - name: "Terraform Apply"
      run: terraform apply -auto-approve
      working-directory: ./clusters/core-services-cluster/terraform/

This ensures that your infrastructure remains consistent and automatically updated whenever changes are pushed to the repository.

Summary

  1. Cluster-specific Terraform code: Each cluster has its own terraform/ folder with infrastructure defined via Terraform for that cluster.
  2. Modular code with centralized modules: Use the