Merge pull request #26 from Ortec-Finance/broker-scaledown-component

Enabling Zone Redundant High Availability Component for Sailfish-broker

CHANGELOG.md

@@ -1,4 +1,10 @@
 # Changelog
+
+## v0.14.0
+Added a new component that allows you to set the Sailfish Broker in High Availability
+This will make your Queue Messages Zone redundant, which means if one zone goes down, the messages will be migrated to another.
+Fixes to observability dashboard outofsync issues 
+
 ## v0.13.0
 Removed `activeDeadlineSeconds` from base configuration as it does not comply with the Job Paradigm as we intend it. Added Documentation that explains how the Job Paradigm is used in Sailfish.
 

docs/features/high-availability.md

@@ -0,0 +1,8 @@
+# High Availabilty
+While Sailfish Runners and Managers already are resilient as they run across multiple availability zones and can spin back up upon failures, The broker does not.
+
+By enabling this component you instead spin up two to three instances of the Sailfish-Broker. The `messageMigration` feature is enabled which will handle the migration of messages in the case of a Broker shutting down in a certain zone.
+
+## Compatibility
+This feature is not compatible with the `ephemeral-broker` as the `messageMigration` feature requires the use of PersistentVolumes.
+

k8s/observability/sailfish-monitoring-dashboard.yaml

@@ -5,7 +5,7 @@ metadata:
   labels:
     app: grafana
 spec:
-  name: sailfish-monitoring
   configMapRef:
     name: sailfish-monitoring-configmap
-    key: dashboard.json
+    key: dashboard.json
+  customFolderName: Sailfish dashboards

k8s/sailfish/base/foundation/broker.yaml

@@ -14,12 +14,11 @@ spec:
   console:
     expose: true
   deploymentPlan:
-    image: placeholder
+    persistenceEnabled: true
+    messageMigration: false
     jolokiaAgentEnabled: false
     journalType: nio
     managementRBACEnabled: true
-    messageMigration: false
-    persistenceEnabled: true
     storage:
       size: 4Gi
       storageClassName: azurefile-csi # is aro specific

k8s/sailfish/components/high-availability/kustomization.yaml

@@ -0,0 +1,53 @@
+kind: Component
+
+patches:
+- target:
+    group: broker.amq.io
+    version: v1beta1
+    kind: ActiveMQArtemis
+    name: sailfish-broker
+  patch: |-
+    - op: replace
+      path: /spec/deploymentPlan/messageMigration
+      value: true
+
+- target:
+    group: broker.amq.io
+    version: v1beta1
+    kind: ActiveMQArtemis
+    name: sailfish-broker
+  patch: |-
+    - op: replace
+      path: /spec/deploymentPlan/size
+      value: 2
+
+- target:
+    group: keda.sh
+    version: v1alpha1
+    kind: ScaledObject
+    name: sailfish-amq-broker-autoscaler
+  patch: |-
+    - op: replace
+      path: /spec/maxReplicaCount
+      value: 3
+
+- target:
+    group: keda.sh
+    version: v1alpha1
+    kind: ScaledObject
+    name: sailfish-amq-broker-autoscaler
+  patch: |-
+    - op: replace
+      path: /spec/minReplicaCount
+      value: 2
+
+- target:
+    group: keda.sh
+    version: v1alpha1
+    kind: ScaledObject
+    name: sailfish-amq-broker-autoscaler
+  patch: |-
+    - op: replace
+      path: /spec/idleReplicaCount
+      value: 0
+

k8s/sailfish/overlay/kustomization.yaml

@@ -4,6 +4,7 @@ resources:
 components:
  - ../components/sailfish-gateway
  - ../components/broker-scale-to-zero
+ - ../components/high-availability
 
 
 

sailfish-example/sailfish-py/gateway/gateway.py

@@ -16,7 +16,7 @@
 Demonstrates how to create an amqp connection and a sender to publish messages.
 """
 class Send(MessagingHandler):
-    def __init__(self, url, address, job, username, password, QoS=1):
+    def __init__(self, url, address, job, username, password, QoS=2):
         super(Send, self).__init__()
 
         # amqp broker host url

sailfish-example/sailfish-py/run-manager/runmanager.py

@@ -70,7 +70,7 @@ def on_disconnected(self, event):
 Demonstrates how to create an amqp connection and a sender to publish messages.
 """
 class Send(MessagingHandler):
-    def __init__(self, url, address, jobs, username, password, QoS=1):
+    def __init__(self, url, address, jobs, username, password, QoS=2):
         super(Send, self).__init__()
 
         # amqp broker host url