init

.gitignore

@@ -0,0 +1,15 @@
+__pycache__
+*.egg-info
+pip-wheel-metadata/
+.eggs
+
+build/
+dist/
+
+.tox
+
+.venv
+
+tests/e2e/log/*.log
+
+src/heatspreader/version.py

.pre-commit-config.yaml

@@ -0,0 +1,15 @@
+repos:
+- repo: https://github.com/pre-commit/pre-commit-hooks
+  rev: v2.3.0
+  hooks:
+  - id: check-yaml
+  - id: check-toml
+  - id: check-json
+  - id: end-of-file-fixer
+  - id: trailing-whitespace
+- repo: https://github.com/python/black
+  rev: 19.3b0
+  hooks:
+  - id: black
+    language_version: python3
+    args: [--check, --diff]

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2019 ACTiCLOUD
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

Makefile

@@ -0,0 +1,18 @@
+clean:
+	find . -name "__pycache__" | xargs rm -rf
+	find . -name "*.pyc" | xargs rm -rf
+	find . -name "*.egg-info" | xargs rm -rf
+	rm -rf ./.tox
+	rm -rf ./.pytest_cache
+	rm -rf ./.eggs
+	rm -rf ./pip-wheel-metadata
+	rm -rf ./dist
+	rm -rf ./build
+	rm ./src/heatspreader/version.py
+
+.PHONY: clean
+
+wheel:
+	python setup.py bdist_wheel
+
+.PHONY: wheel

README.md

@@ -0,0 +1,130 @@
+Heat Spreader
+=============
+
+Heat Spreader controls OpenStack Heat stacks across multiple clouds.
+
+Currently, it specifically controls a configurable count parameter on
+templates that use a resource of type OS::Heat::ResourceGroup. It does this to
+scale the number of instances a resource group should run in each cloud.
+
+# Configuration
+
+## OpenStack cloud configuration
+
+The first thing you need to do is configure the different clouds that you want
+to control. Heat Spreader uses the OpenStack SDK, read the
+[configuration guide](https://docs.openstack.org/openstacksdk/latest/user/config/configuration.html)
+on how to configure the cloud authentication.
+
+## Heat Spreader configuration
+
+When the clouds are configured the next step is to configure the Heat Spreader
+configuration. The following configuration options are available:
+
+* backend - Configuration for how the state should be persisted.
+* clouds - A list of clouds that should be controlled.
+* server - HTTP server configuration.
+
+See the [example configurations](./examples) for a sample of a server and a
+client configuration.
+
+Heat Spreader, by default, looks for the config file at
+`${HOME}/.config/openstack/heat-spreader.yaml`, to use a different config
+file path set the environment variable `HEAT_SPREADER_CONFIG_FILE`.
+
+# Usage
+
+## Quickstart
+
+**Start the HTTP API server and the scaling controller loop**
+
+```
+heat-spreader run
+```
+
+**Add a multicloud stack**
+
+```
+heat-spreader add [stack name] \
+    --count [desired count] --parameter [stack count parameter]
+```
+
+**Add multicloud stack cloud weights**
+
+```
+heat-spreader weight set [stack name] [cloud 1 name] --weight 0.5
+heat-spreader weight set [stack name] [cloud 2 name] --weight 0.5
+```
+
+## Multicloud scaling
+
+### Weights
+
+When the desired count is calculated the total count is multiplied by the
+weight each cloud has been configured with. For example, if a multicloud stack
+configuration looks like this:
+```
+count: 10
+cloud_1 weight: 0.6
+cloud_2 weight: 0.2
+cloud_3 weight: 0.2
+```
+The resulting count in each cloud's stack will be:
+```
+cloud_1 count: 6
+cloud_2 count: 2
+cloud_3 count: 2
+```
+
+### Failover
+
+Whenever a cloud is unreachable the configured weight for that cloud will be
+spread across the other reachable clouds evenly. For example, if a multicloud
+stack configuration is the following:
+```
+count: 10
+cloud_1 weight: 0.6
+cloud_2 weight: 0.2
+cloud_3 weight: 0.2
+```
+And `cloud_3` becomes unreachable, the resulting weights will become:
+```
+cloud_1 weight: 0.7
+cloud_2 weight: 0.3
+cloud_3 weight: 0.0
+```
+Which finally will make the count parameter in each stack to be updated to:
+```
+cloud_1 count: 7
+cloud_2 count: 3
+cloud_3 count: 0
+```
+Essentially rescheduling the resources lost due to the cloud being down.
+
+# Q&A
+
+Q: Why not simply use the remote stack feature in Heat, i.e. a stack resource
+with region and credential attributes set?
+
+A: The goals of this project and remote stacks in Heat are different. While
+using remote stacks are great for creating and managing stacks in a remote
+region from a local Heat deployment this project aims to view multiple remote
+stacks as a single unit working in a collaborative fashion across multiple
+clouds/regions.
+
+One example of this is how the Heat Spreader controller reacts to a cloud no
+longer being reachable by rescheduling the lost instances to any of the
+cloud(s) that are still active.
+
+# Limitations
+
+## False-positive cloud failure detection
+
+Currently the health checking mechanism is based on internal exceptions, e.g.
+whether the scaling controller can communicate with Heat or not. Since a
+connection failure can occur due to a lot of different reasons and not only an
+actual cloud outage this could lead to an overcommit of resources until the
+connection is again established.
+
+This could be mitigated by implementing support for some form of external
+distributed watchdog system.

examples/client_config.yaml

@@ -0,0 +1,4 @@
+backend:
+  type: remote
+  host: localhost
+  port: 8080

examples/server_config.yaml

@@ -0,0 +1,9 @@
+backend:
+  type: sqlite
+  database: "/tmp/actimanager_multicloud_db"
+clouds:
+  - athens
+  - manchester
+server:
+  address: 127.0.0.1
+  port: 8080

pyproject.toml

@@ -0,0 +1,2 @@
+[tool.black]
+line-length = 79

pytest.ini

@@ -0,0 +1,9 @@
+[pytest]
+filterwarnings =
+    ; Haven't seen a patch yet
+    ; https://github.com/openstack/openstacksdk/blob/master/openstack/resource.py
+    ignore::DeprecationWarning:openstack.resource
+    ; Fixed in https://github.com/pallets/jinja/pull/867
+    ; Waiting for new release
+    ignore::DeprecationWarning:jinja2.runtime
+    ignore::DeprecationWarning:jinja2.utils

setup.py

@@ -0,0 +1,36 @@
+from setuptools import setup, find_packages
+
+setup(
+    name="heat-spreader",
+    use_scm_version={"write_to": "src/heatspreader/version.py"},
+    setup_requires=["setuptools_scm"],
+    description="Heat multicloud service",
+    author="ACTiCLOUD",
+    author_email="[email protected]",
+    url="https://acticloud.eu",
+    license="MIT",
+    classifiers=[
+        "Development Status :: 3 - Alpha",
+        "Programming Language :: Python :: 3",
+        "Programming Language :: Python :: 3.7",
+    ],
+    packages=find_packages(where="src"),
+    package_dir={"": "src"},
+    install_requires=[
+        "setuptools_scm>=3.3.1,<4.0.0",
+        "aiohttp>=3.6.0,<4.0.0",
+        "aiohttp-apispec>=1.3.0,<2.0.0",
+        "colorama",
+        "marshmallow>=3.2.0,<4.0.0",
+        "marshmallow-oneofschema>=2.0.1,<3.0.0",
+        "openstacksdk>=0.31.1",
+        "peewee>=3.10.0,<4.0.0",
+        "python-heatclient",
+        "pyyaml>=5.1.2,<6.0.0",
+        "structlog>=19.1.0,<20.0.0",
+    ],
+    extras_require={"test": ["tox"]},
+    entry_points={
+        "console_scripts": ["heat-spreader = heatspreader.shell.__main__:main"]
+    },
+)

src/heatspreader/__init__.py

@@ -0,0 +1,16 @@
+from .client import Client, WeightNotFound
+from .config import Config, RemoteBackendConfig, SqliteBackendConfig
+from .log import setup_logging
+from .state import MulticloudStack
+from .store import MulticloudStackNotFound
+
+__all__ = [
+    "Client",
+    "Config",
+    "MulticloudStack",
+    "MulticloudStackNotFound",
+    "RemoteBackendConfig",
+    "setup_logging",
+    "SqliteBackendConfig",
+    "WeightNotFound",
+]

src/heatspreader/client/__init__.py

@@ -0,0 +1,4 @@
+from .client import Client
+from .exceptions import WeightNotFound
+
+__all__ = ["Client", "WeightNotFound"]

src/heatspreader/client/client.py

@@ -0,0 +1,75 @@
+from ..store import MulticloudStackStore
+from ..state import MulticloudStack
+
+from .exceptions import WeightNotFound
+
+
+class Client:
+    def __init__(self, config):
+        self.config = config
+
+        self._store = MulticloudStackStore(config)
+
+    async def __aenter__(self):
+        return self
+
+    async def __aexit__(self, exc_type, exc_val, exc_tb):
+        await self.close()
+
+    async def close(self):
+        await self._store.close()
+
+    async def create(self, stack_name, count, count_parameter, weights={}):
+        multicloud_stack = MulticloudStack(
+            stack_name=stack_name,
+            count=count,
+            count_parameter=count_parameter,
+            weights=weights,
+        )
+
+        await self._store.set(multicloud_stack)
+
+        return multicloud_stack
+
+    async def get(self, stack_name):
+        return await self._store.get(stack_name)
+
+    async def update(self, stack_name, count=None, count_parameter=None):
+        multicloud_stack = await self._store.get(stack_name)
+
+        if count is not None:
+            multicloud_stack.count = count
+
+        if count_parameter is not None:
+            multicloud_stack.count_parameter = count_parameter
+
+        await self._store.set(multicloud_stack)
+
+        return multicloud_stack
+
+    async def delete(self, stack_name):
+        await self._store.delete(stack_name)
+
+    async def list(self):
+        return await self._store.list()
+
+    async def weight_set(self, stack_name, cloud_name, weight):
+        multicloud_stack = await self._store.get(stack_name)
+
+        multicloud_stack.weights[cloud_name] = float(weight)
+
+        await self._store.set(multicloud_stack)
+
+        return multicloud_stack
+
+    async def weight_unset(self, stack_name, cloud_name):
+        multicloud_stack = await self._store.get(stack_name)
+
+        try:
+            del multicloud_stack.weights[cloud_name]
+        except KeyError:
+            raise WeightNotFound(stack_name, cloud_name)
+
+        await self._store.set(multicloud_stack)
+
+        return multicloud_stack

src/heatspreader/client/exceptions.py

@@ -0,0 +1,6 @@
+class WeightNotFound(Exception):
+    def __init__(self, stack_name, cloud_name):
+        super().__init__(
+            f"Weight for cloud '{cloud_name}' not found in multicloud stack: "
+            f"{stack_name}"
+        )