Add acceptance tests for how provider handles billing_project arguments

[SarahFrench]

This PR adds acceptance tests for usage of billing_project that demonstrate:

• how the provider behaves when provider configuration arguments come from different sources ( config vs ENVs)
• schema-level validation that's in place, e.g. handling of empty arrays
• use cases: how does this argument impact the providers behaviour in plan/apply
  • I've tested the use of billing_project + user_project_override=false to show that billing_project has no impact then
  • I've tested the use of billing_project + user_project_override=true to show that billing_project does have impact then

NOTE: I added some usage tests for the PF provider, but they should be replaced in future because being limited to Firebase data sources to test the PF provider makes the test very awkward

Release Note Template for Downstream PRs (will be copied)

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 316 insertions(+))
google-beta provider: Diff ( 2 files changed, 316 insertions(+))

[SarahFrench]

Hi @NickElliot - I've currently stuck with something in this PR that seems related to the VCR test project, or specifically to tests running on PRs. It's odd!

The TestAccSdkProvider_billing_project test that's failing here is failing because the test expects errors but doesn't experience any.

However there should be an error, as the test is structured to cause one. When the SDK test is run in TeamCity (using the VCR project) the test passes as expected.

Any idea what could be causing this? I'm stuck... the solution could just be to skip in VCR?

[github-actions]

@NickElliot This PR has been waiting for review for 3 weekdays. Please take a look! Use the label disable-review-reminders to disable these notifications.

[github-actions]

@GoogleCloudPlatform/terraform-team @NickElliot This PR has been waiting for review for 1 week. Please take a look! Use the label disable-review-reminders to disable these notifications.

[github-actions]

@GoogleCloudPlatform/terraform-team @NickElliot This PR has been waiting for review for 2 weeks. Please take a look! Use the label disable-review-reminders to disable these notifications.

[SarahFrench]

@NickElliot , could you please take a look at this PR? If you're at capacity with work I can reassign, just let me know!

Similar for #11686

[SarahFrench]

Hey @c2thorn, here's the PR where I need a bit of help figuring out why tests behave differently when I run them locally vs on this PR. There's a message summarising the issue here : #11610 (comment)

[c2thorn]

tentative lgtm depending on how you want to address https://github.com/GoogleCloudPlatform/magic-modules/pull/11610/files#r1811341541

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 503 insertions(+))
google-beta provider: Diff ( 2 files changed, 635 insertions(+))

[modular-magician]

Tests analytics

Total tests: 4209
Passed tests: 3786
Skipped tests: 420
Affected tests: 3

Click here to see the affected service packages

All service packages are affected

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

• TestAccBackupDRBackupVault_fullUpdate
• TestAccSdkProvider_billing_project
• TestAccSqlDatabaseInstance_Edition_Downgrade

Get to know how VCR tests work

[modular-magician]

🟢 Tests passed during RECORDING mode:
TestAccBackupDRBackupVault_fullUpdate [Debug log]

🟢 No issues found for passed tests after REPLAYING rerun.

---

🔴 Tests failed during RECORDING mode:
TestAccSdkProvider_billing_project [Error message] [Debug log]
TestAccSqlDatabaseInstance_Edition_Downgrade [Error message] [Debug log]

🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR.

View the build log or the debug log for each test

[SarahFrench]

provider_billing_project_test.go:200: Step 2/3, expected an error but got none

I'd missed that 2 tests are failing in TestAccSdkProvider_billing_project. I'll confirm that it passes ok in the nightlies and then make it skip in VCR too. Edit: confirmed.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 507 insertions(+))
google-beta provider: Diff ( 2 files changed, 639 insertions(+))

[modular-magician]

Tests analytics

Total tests: 4209
Passed tests: 3788
Skipped tests: 420
Affected tests: 1

Click here to see the affected service packages

All service packages are affected

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

• TestAccSqlDatabaseInstance_Edition_Downgrade

Get to know how VCR tests work

[modular-magician]

🔴 Tests failed during RECORDING mode:
TestAccSqlDatabaseInstance_Edition_Downgrade [Error message] [Debug log]

🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR.

View the build log or the debug log for each test

[SarahFrench]

Going ahead with merge!

[SarahFrench]

Ah dammit, it failed in the nightlies: https://hashicorp.teamcity.com/buildConfiguration/TerraformProviders_GoogleCloud_GOOGLE_NIGHTLYTESTS_GOOGLE_PACKAGE_PROVIDER/254854?branch=refs%2Fheads%2Fnightly-test&buildTypeTab=overview&mode=builds&hideTestsFromDependencies=false&buildTab=tests&expandedTest=build%3A%28id%3A254854%29%2Cid%3A2000000015

I'll ask in the team channel about how to use billing_project and user_project_override together

[SarahFrench]

For completeness, noting here:

The tests failed in VCR due to hashicorp/terraform-provider-google#20019
When the test runs in VCR mode the unaliased provider is the first to be configured, so its Config struct is cached. Later when the aliased provider is used in the test the provider arguments are effectively ignored because the aliased provider pulls in the Config of the un-aliased provider (which was configured with different arguments).

This is why the test passes when run outside of VCR mode and failed in VCR mode.

The failure identified here is just bad luck and unrelated to the VCR-related confusion!