Cloud Resources Refactor

[bobbyiliev]

I've been thinking with #5 based on the doc here

Each resource within our provider will be able to utilize one of these clients depending on its requirements. For example, a resource that manages database configurations will use the DBClient, whereas another that interacts with Frontegg services will use FronteggClient and etc.

struct {
	DB       *clients.DBClient
	Frontegg *clients.FronteggClient
}

This PR is more of a PoC so we could start a further discussion, it implements a materialize_app_password resource that utilizes the FrontEgg API:

resource "materialize_app_password" "example_password" {
  name = "tf-example-app-password2"
}

List of things that need to be done to get this PR merged:

• Add a design doc
• Add Frontegg Client
• Add Tests for the Frontegg Client
• Add Cloud API Client
• Add Database Client
• Add Cloud API Resources
• Add Tests for the Cloud API Client
• Refactor the Provider configuration to use the new clients
• Refactor all existing database resources to use the client
• Refactor all existing tests to use the client
• Add the new Cloud Resources:
  • materialize_app_password
  • materialize_user
• Add the new data sources:
  • materialize_regions

Refactor resources to use the new DB client:

• Cluster Replicas
• Clusters
• Connection - AWS Privatelink
• Connection - Confluent Schema Registry
• Connection - Kafka
• Connection - Postgres
• Connection - SSH
• Databases
• Grants
• Grants Default
• Roles
• Indexes
• Materialized Views
• Schemas
• Secrets
• Sinks - Kafka
• Sources - Kafka
• Sources - Load Generator
• Sources - Postgres
• Sources - Webhook
• Tables
• Types
• Views

Tasks that can be contributed separately in follow-up PRs:

• Add the new Cloud Resources:
  • materialize_region
• Add the new data sources:
  • materialize_organization

[benesch]

Nice, this makes sense to me! Note that you probably want to support multiple "DB" clients, one per region:

type Region string

const (
        awsUsEast1 Region = "aws/us-east-1"
        awsUsWest2 Region = "aws/us-west-2"
)

struct {
	DB       map[Region]*clients.DBClient
	Frontegg *clients.FronteggClient
}

[dehume]

This looks great. Exciting to add these

[dehume]

Is the admin endpoint unique to each organization's Materialize account?

[benesch]

No, but you'll need some way to override it for tests. Production and staging have different admin endpoints, for example.

[CLAassistant]

All committers have signed the CLA.

[benesch]

Looking good! Dropped in a few comments.

[benesch]

Nice! I only took a very quick skim, but this all seems to be headed in a wonderful direction. 👍🏽

[sjwiesman]

Can you say more about why we need the deep copy?

[bobbyiliev]

This is primarily related to concurrency and the immutability of the original request. As http.Request.Body can only be read once, a new body needs to be copied:

https://stackoverflow.com/questions/62017146/http-request-clone-is-not-deep-clone

Will try to adjust the comment to give a bit more details!

[sjwiesman]

How about you include the stack overflow link in the comment.

[bobbyiliev]

Sounds good, just added the link in the func comments.