convert to accretion-style deployment sets

[mattsb42-aws]

In tackling the same problem of "how do I manage several related stacks" multiple times from multiple different angles, I have come to really like the general pattern I arrived at with Accretion[1].

To summarize the general pattern: there are several stacks that in total constitute an application deployment. These stacks are created with random stack names and a record of the stack names and associated metadata is stored in a "deployment file".

Other approaches that I have investigated include a reliance on the caller to provide unique stack names the definition of a meta-stack that contains all related stacks under it as nested stacks. Pipeformer currently uses the later approach.

Unique Stack Names

This is the simplest and most straightforward approach, and because of this is probably the most commonly used. When tools take this approach, they do have the benefit of very explicitly identifying the application that deployed the stack, but they create some very important downsides. Most importantly, it makes it very difficult for multiple instances of an application to be deployed in the same account in the same region.

This is not an issue if the application in question should only ever have a single deployment per account/region, but in cases of things like pipeformer, where you might reasonably want dozens if not hundreds of deployments in the same account/region, this becomes problematic.

You can work around the name conflict issue to an extent by relying on the user to define a unique name, but this relies on the user to come up with unique names. Again, at the scale that I plan for pipeformer to be used, this is not a desirable pattern.

Meta-Stack

The meta-stack pattern seems like a good idea on the surface, but in working through how to maintain meta-stack trees, I have come to the conclusion that they are more trouble than they are worth.

The general idea of the meta-stack is that you have a single parent stack, that contains all other stacks as nested stacks, possibly nested arbitrarily deeply.

This creates several problems, not least of which being that bootstrapping the stack tree cleanly requires either a tiered deployment or a lot of overly clever jockeying with wait conditions.

The meta-stack pattern becomes even more complex when you look at updating a single nested stack. Because of the nested structure, every parent stack all the way up must also be updated. This can create significant complexity, especially if the target stack is more than one layer deep, and runs the risk introducing side-effects on stacks that should not have been modified.

The meta-stack pattern also makes it much more complicated to split the nested stacks up into individual components. #41

Deployment File

The deployment file pattern takes a much simpler approach by making one requirement: stack identity and relationships are defined and stored out of band.

In the accretion example, this takes the form of a file on-disk with the idea that you will then store that file in your source control system. However this data could instead be stored in a database if a more centralized mechanism is desired.

Because the stack identities and relationships are stored outside of CloudFormation, we can use random, unique, values for the stack names. This means that we can deploy arbitrarily many instances of a single application in the same account/region.

Because all stacks are maintained independently, we can define weak dependencies between them rather than hard dependencies. This means that we can much more easily separate them and deploy groups or individual stacks independently. This also means that we can handle unusual update stories (ex: for the inputs stack) independently and distinctly from all other stacks.

Some additions that I think we should make to this pattern are:

1. Rather than each stack having a completely independent random component, each deployment should have a unique ID that is used, along with the logical stack name, to construct the physical stack name.
2. We should add tags to the stacks that identify their role, application name, and deployment ID. Tying an application name lets us provide a human-friendly relationship while the deployment ID lets us make and uniquely identify separate deployments, even if they have the same application name.

[1] https://accretion.readthedocs.io/en/latest/src/cli.html#deployment-file

[mattsb42-aws]

Closing down project: #52