Merge branch 'GoogleCloudPlatform:main' into prs/add-support-for-shie…

…ldedInstanceInitialState

.ci/magician/github/membership_data.go

@@ -116,5 +116,10 @@ var (
 			startDate: newDate(2024, 11, 1, pdtLoc),
 			endDate:   newDate(2024, 11, 11, pdtLoc),
 		},
+		{
+			id:        "shuyama1",
+			startDate: newDate(2024, 11, 26, pdtLoc),
+			endDate:   newDate(2024, 12, 4, pdtLoc),
+		},
 	}
 )

GNUmakefile

@@ -75,7 +75,7 @@ tpgtools:
 clean-provider:
 	cd $(OUTPUT_PATH);\
 		go mod download;\
-		find . -type f -not -wholename "./.git*" -not -wholename "./.changelog*" -not -name ".travis.yml" -not -name ".golangci.yml" -not -name "CHANGELOG.md" -not -name "CHANGELOG_v*.md" -not -name "GNUmakefile" -not -name "docscheck.sh" -not -name "LICENSE" -not -name "README.md" -not -wholename "./examples*" -not -name ".go-version" -not -name ".hashibot.hcl" -print0 | xargs -0 git rm > /dev/null
+		find . -type f -not -wholename "./.git*" -not -wholename "./.changelog*" -not -name ".travis.yml" -not -name ".golangci.yml" -not -name "CHANGELOG.md" -not -name "CHANGELOG_v*.md" -not -name "GNUmakefile" -not -name "docscheck.sh" -not -name "LICENSE" -not -name "CODEOWNERS" -not -name "README.md" -not -wholename "./examples*" -not -name ".go-version" -not -name ".hashibot.hcl" -print0 | xargs -0 git rm > /dev/null
 
 clean-tgc:
 	cd $(OUTPUT_PATH);\

docs/content/_index.md

@@ -1,12 +1,16 @@
 ---
 title: "Overview"
+weight: 10
+aliases:
+  - /docs/how-to/types-of-resources
+  - /how-to/types-of-resources
+  - /get-started/how-magic-modules-works
 ---
 
-
 # Magic Modules
 
 Magic Modules is a code generator and CI system that's used to develop the Terraform providers
-for Google Platform, [`google`](https://github.com/hashicorp/terraform-provider-google) (or TPG) and
+for Google Cloud, [`google`](https://github.com/hashicorp/terraform-provider-google) (or TPG) and
 [`google-beta`](https://github.com/hashicorp/terraform-provider-google-beta) (or TPGB).
 
 Magic Modules allows contributors to make changes against a single codebase and develop both
@@ -15,13 +19,97 @@ provider versions simultaneously. After sending a pull request against this repo
 complete output, running presubmit tests, and updating the providers following your
 change.
 
-## Getting started
+## How Magic Modules works
+
+Magic Modules can be thought of as a source of truth for how to map a Google Cloud API resource
+representation to a Terraform resource (or datasource) representation. Magic Modules uses that mapping
+(and additional handwritten code where necessary) to generate "downstream" repositories - in particular,
+the Terraform providers for Google Cloud: [`google`](https://github.com/hashicorp/terraform-provider-google)
+(or TPG) and [`google-beta`](https://github.com/hashicorp/terraform-provider-google-beta) (or TPGB).
+
+Generation of the downstream repositories happens for every new commit in a PR (to a temporary branch owned by
+the [`modular-magician`](https://github.com/modular-magician/) robot user) and on every merge into the main branch
+(to the main branch of downstreams). Generation for PR commits allows contributors to manually examine the changes,
+as well as allowing automatic running of unit tests, acceptance tests, and automated checks such as breaking change
+detection.
+
+### Resource types
+
+There are three types of resources supported by Magic Modules:
+
++ MMv1
++ Handwritten
++ DCL/tpgtools
+
+The following sections describe these tools in detail.
+
+#### MMv1
+
+MMv1 consists of a set of "products"; each product contains one or more "resources".
+
+Each product has a folder in
+[`magic-modules/mmv1/products`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/products).
+The name of the folder is the "product name", which usually corresponds to the API subdomain covered by the
+product (such as `compute.googleapis.com`). Each product folder contains a product configuration file
+(`product.yaml`) and one or more resource configuration files (`ResourceName.yaml`). The actual name of a
+`ResourceName.yaml` file usually matches the name of a GCP API resource in the product's subdomain.
+
+MMv1 resource configurations may reference handwritten code stored in
+[`magic-modules/mmv1/templates/terraform`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/templates/terraform),
+which will be injected into the generated resource file. Many MMv1 resources also have one or more handwritten tests,
+which are stored in the appropriate service folder inside
+[`magic-modules/mmv1/third_party/terraform/services`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/terraform/services).
+
+In the providers, MMv1-based resources are stored in `PROVIDER/services/PRODUCT/resource_PRODUCT_RESOURCE.go`, where `PROVIDER`
+is `google` or `google-beta`, `PRODUCT` is the product name, and RESOURCE is the GCP API resource's name converted to
+[snake case ↗](https://en.wikipedia.org/wiki/Snake_case).
+
+MMv1-based files start with the following header:
+
+```
+***     AUTO GENERATED CODE    ***    Type: MMv1     ***
+```
+
+#### Handwritten
+
+Handwritten resources and datasources are technically part of MMv1; however, they are not generated from YAML configurations.
+Instead, they are written as Go code with minimal go template "version guards" to exclude beta-only features from the `google`
+provider.
+
+Handwritten resources and datasources can be grouped by "service", which generally corresponds to the API subdomain the resource
+or datasource interacts with.
+
+In addition to the core implementation, handwritten resources and datasources will also have documentation, tests, and sweepers
+(which clean up stray resources left behind by tests). Each type of code is stored in the following locations:
+
++ Resource & datasource implementation: In the appropriate service folder inside
+  [`magic-modules/mmv1/third_party/terraform/services`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/terraform/services)
++ Resource documentation:
+  [`magic-modules/mmv1/third_party/terraform/website/docs/r`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/terraform/website/docs/r)
++ Datasource documentation:
+  [`magic-modules/mmv1/third_party/terraform/website/docs/d`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/terraform/website/docs/d)
++ Tests: In the appropriate service folder inside
+  [`magic-modules/mmv1/third_party/terraform/services`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/terraform/services)
++ Sweepers: [`magic-modules/mmv1/third_party/terraform/utils`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/terraform/utils)
+
+In the providers, handwritten resources and datasources are stored in `PROVIDER/services/SERVICE/FILENAME.go`, where `PROVIDER`
+is `google` or `google-beta`, `SERVICE` is the service name, and `FILENAME` is the name of the handwritten file in magic-modules.
+Handwritten files do not have an `AUTO GENERATED CODE` header.
+
+#### DCL aka tpgtools (maintenance mode)
+
+DCL / tpgtools is similar to MMv1; however, it is in maintenance mode, which means that new resources using the DCL are not being added.
+
+DCL-based files start with the following header:
+
+```
+***     AUTO GENERATED CODE    ***    Type: DCL     ***
+```
 
-Check out the [setup guide]({{< ref "/get-started/generate-providers" >}}) for information on how to set up your environment.
 
 ## Other Resources
 
-* [Extending Terraform](https://www.terraform.io/plugin)
-   * [How Terraform Works](https://www.terraform.io/plugin/how-terraform-works)
-   * [Writing Custom Providers / Calling APIs with Terraform Providers](https://learn.hashicorp.com/collections/terraform/providers?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS)
-* [Terraform Glossary](https://www.terraform.io/docs/glossary)
++ [Extending Terraform](https://www.terraform.io/plugin)
+   + [How Terraform Works](https://www.terraform.io/plugin/how-terraform-works)
+   + [Writing Custom Providers / Calling APIs with Terraform Providers](https://learn.hashicorp.com/collections/terraform/providers?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS)
++ [Terraform Glossary](https://www.terraform.io/docs/glossary)

docs/content/get-started/contribution-process.md → docs/content/contribution-process.md

@@ -1,14 +1,17 @@
 ---
 title: "Contribution process"
-weight: 50
+weight: 11
 aliases:
   - /docs/getting-started/contributing
   - /getting-started/contributing
   - /get-started/contributing
+  - /get-started/contribution-process
 ---
 
 # Contribution process
 
+This page explains how you can contribute code and documentation to the	`magic-modules` repository.
+
 ## Before you begin
 
 1. Familiarize yourself with [GitHub flow](https://docs.github.com/en/get-started/quickstart/github-flow)
@@ -19,14 +22,15 @@ aliases:
 
 ## Contribute code
 
+1. [Set up your development environment]({{< ref "/develop/set-up-dev-environment" >}})
 1. [Create a new branch for your change](https://docs.github.com/en/get-started/quickstart/github-flow#create-a-branch)
 1. Make the code change. For example:
    - [Add or modify a resource]({{< ref "/develop/resource" >}})
    - [Add resource tests]({{< ref "/develop/test/test" >}})
    - [Add a datasource]({{< ref "/develop/add-handwritten-datasource" >}})
    - [Promote to GA]({{< ref "/develop/promote-to-ga" >}})
    - [Make a breaking change]({{< ref "/develop/breaking-changes/make-a-breaking-change" >}})
-1. [Generate the providers]({{< ref "/get-started/generate-providers" >}}) that include your change.
+1. [Generate the providers]({{< ref "/develop/generate-providers" >}}) that include your change.
 1. [Run provider tests locally]({{< ref "/develop/test/run-tests" >}}) that are relevant to the change you made
 1. [Create a pull request (PR)]({{< ref "/contribute/create-pr" >}})
 1. Make changes in response to [code review]({{< ref "/contribute/create-pr#code-review" >}})

docs/content/develop/add-handwritten-datasource.md

@@ -1,7 +1,7 @@
 ---
 title: "Add a datasource"
 summary: "Datasources are like terraform resources except they don't *create* anything."
-weight: 40
+weight: 14
 aliases:
   - /docs/how-to/add-handwritten-datasource
   - /how-to/add-handwritten-datasource
@@ -52,5 +52,5 @@ library, or the raw HTTP client used in MMV1 through `SendRequest`.
 1. Open the data source documentation in [`magic-modules/third_party/terraform/website/docs/d/`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/terraform/website/docs/d) using an editor of your choice.
    - The name of the file is the name of the data source without a `google_` prefix. For example, for `google_compute_instance`, the file is called `compute_instance.html.markdown`
 2. Modify the documentation as needed according to [Handwritten documentation style guide]({{< ref "/develop/handwritten-docs-style-guide" >}}).
-4. [Generate the providers]({{< ref "/get-started/generate-providers.md" >}})
+4. [Generate the providers]({{< ref "/develop/generate-providers" >}})
 5. Copy and paste the generated documentation into the Hashicorp Registry's [Doc Preview Tool](https://registry.terraform.io/tools/doc-preview) to see how it is rendered.

docs/content/develop/breaking-changes/make-a-breaking-change.md

@@ -174,7 +174,7 @@ or resource. It is also great to log warnings at runtime if possible.
 When working on your breaking change, make sure that your base branch
 is `FEATURE-BRANCH-major-release-{{% param "majorVersion" %}}`. This
 means that you will follow the standard
-[contribution process]({{< ref "/get-started/contribution-process" >}})
+[contribution process]({{< ref "/contribution-process" >}})
 with the following changes:
 
 1. Before you start, check out and sync your local `magic-modules` and provider

docs/content/develop/custom-code.md

@@ -1,11 +1,11 @@
 ---
 title: "Add custom resource code"
-weight: 39
+weight: 15
 ---
 
 # Add custom resource code
 
-This document covers how to add "custom code" to [MMv1 resources]({{< ref "/get-started/how-magic-modules-works#mmv1" >}}). Custom code can be used to add arbitrary logic to a resource while still generating most of the code; it allows for a balance between maintainability and supporting real-worlds APIs that deviate from what MMv1 can support. Custom code should only be added if the desired behavior can't be achieved otherwise.
+This document covers how to add "custom code" to [MMv1 resources]({{< ref "/#mmv1" >}}). Custom code can be used to add arbitrary logic to a resource while still generating most of the code; it allows for a balance between maintainability and supporting real-worlds APIs that deviate from what MMv1 can support. Custom code should only be added if the desired behavior can't be achieved otherwise.
 
 Most custom code attributes are strings that contain a path to a template file relative to the `mmv1` directory. For example:
 
@@ -17,7 +17,7 @@ custom_code:
 
 By convention, the template files are stored in a directory matching the type of custom code, and the name of the file includes the resource (and, if relevant, field) impacted by the custom code. Like handwritten resource and test code, custom code is written as go templates which render go code.
 
-When in doubt about the behavior of custom code, write the custom code, [generate the providers]({{< ref "/get-started/generate-providers" >}}), and inspect what changed in the providers using `git diff`.
+When in doubt about the behavior of custom code, write the custom code, [generate the providers]({{< ref "/develop/generate-providers" >}}), and inspect what changed in the providers using `git diff`.
 
 The following sections describe types of custom code in more detail.
 

docs/content/develop/diffs.md

@@ -1,6 +1,6 @@
 ---
 title: "Fix diffs"
-weight: 60
+weight: 18
 aliases:
   - /develop/permadiff
 ---

docs/content/develop/generate-providers.md

@@ -0,0 +1,96 @@
+---
+title: "Generate the providers"
+weight: 19
+aliases:
+  - /docs/getting-started/setup
+  - /getting-started/setup
+  - /docs/getting-started/generate-providers
+  - /getting-started/generate-providers
+  - /get-started/generate-providers
+---
+
+# Generate `google` and `google-beta` providers
+
+After making a change to the Terraform providers for Google Cloud, you must
+integrate your changes with the providers. This page explains how to generate
+provider changes to the `google` and `google-beta` Terraform providers.
+
+## Before you begin
+
+1. [Set up your development environment]({{< ref "/develop/set-up-dev-environment" >}}).
+1. Update `magic-modules` as needed. These updates could be any of the following changes:
+  + [Adding or modifying a resource]({{< ref "/develop/resource" >}}).
+  + [Adding a datasource]({{< ref "/develop/add-handwritten-datasource" >}}).
+  + [Adding custom resource code]({{< ref "/develop/custom-code" >}}).
+  + [Promoting a resource to GA]({{< ref "/develop/promote-to-ga" >}}).
+
+## Generate a provider change
+
+1. Clone the `google` and `google-beta` provider repositories with the following commands:
+
+   ```bash
+   git clone https://github.com/hashicorp/terraform-provider-google.git $GOPATH/src/github.com/hashicorp/terraform-provider-google
+   git clone https://github.com/hashicorp/terraform-provider-google-beta.git $GOPATH/src/github.com/hashicorp/terraform-provider-google-beta
+   ```
+1. Generate changes for the `google` provider:
+   ```bash
+   make provider VERSION=ga OUTPUT_PATH="$GOPATH/src/github.com/hashicorp/terraform-provider-google" PRODUCT=[PRODUCT_NAME]
+   ```
+    Where `[PRODUCT_NAME]` is one of the folder names in
+    https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/products.
+
+    For example, if your product is `bigqueryanalyticshub`, the command would be
+    the following:
+
+     ```bash
+     make provider VERSION=ga OUTPUT_PATH="$GOPATH/src/github.com/hashicorp/terraform-provider-google" PRODUCT=bigqueryanalyticshub
+     ```
+
+1. Generate changes for the `google-beta` provider:
+   ```bash
+   make provider VERSION=beta OUTPUT_PATH="$GOPATH/src/github.com/hashicorp/terraform-provider-google-beta" PRODUCT=[PRODUCT_NAME]
+   ```
+
+    Where `[PRODUCT_NAME]` is one of the folder names in https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/products.
+
+    For example, if your product name is `bigqueryanalyticshub`, the command would be the following:
+
+     ```bash
+     make provider VERSION=beta OUTPUT_PATH="$GOPATH/src/github.com/hashicorp/terraform-provider-google-beta" PRODUCT=bigqueryanalyticshub
+     ```
+
+1. Confirm that the expected changes were generated:
+   ```bash
+   cd $GOPATH/src/github.com/hashicorp/terraform-provider-google
+   git diff -U0
+   cd $GOPATH/src/github.com/hashicorp/terraform-provider-google-beta
+   git diff -U0
+   ```
+
+
+   {{< hint info >}}
+   **Note**: There may be additional changes present due to specifying a
+   `PRODUCT=` value or due to the `magic-modules` repository being out of sync
+   with the provider repositories.
+   {{< /hint >}}
+
+
+## Troubleshoot
+
+### Too many open files {#too-many-open-files}
+
+If you are getting “Too many open files” ulimit needs to be raised.
+
+{{< tabs "ulimit" >}}
+{{< tab "Mac OS" >}}
+```bash
+ulimit -n 8192
+```
+{{< /tab >}}
+{{< /tabs >}}
+
+## What's next
+
++ [Learn how to add resource tests]({{< ref "/develop/test/test" >}})
++ [Learn how to run tests]({{< ref "/develop/test/run-tests" >}})
++ [Learn about `make` commands]({{< ref "/reference/make-commands" >}})

docs/content/develop/promote-to-ga.md

@@ -1,6 +1,6 @@
 ---
 title: "Promote to GA"
-weight: 50
+weight: 16
 ---
 
 # Promote from beta to GA
@@ -9,11 +9,11 @@ This document describes how to promote an existing resource or field that uses M
 
 Handwritten code (including `custom_code`) commonly uses "version guards" in the form of `{{- if ne $.TargetVersionName "ga" }}...{{- end }}` to wrap code that is beta-specific, which need to be removed during promotion.
 
-For more information about types of resources and the generation process overall, see [How Magic Modules works]({{< ref "/get-started/how-magic-modules-works.md" >}}).
+For more information about types of resources and the generation process overall, see [How Magic Modules works]({{< ref "/" >}}).
 
 ## Before you begin
 
-1. Complete the [Generate the providers]({{< ref "/get-started/generate-providers" >}}) quickstart to set up your environment and your Google Cloud project.
+1. Complete the steps in [Set up your development environment]({{< ref "/develop/set-up-dev-environment" >}}) to set up your environment and your Google Cloud project.
 2. Ensure that your `magic-modules`, `terraform-provider-google`, and `terraform-provider-google-beta` repositories are up to date.
    ```
    cd ~/magic-modules