Bug in "external" data provider: Cannot extract value from python-produced json, also extracted value is neither blank nor non-blank

[hashibot]

This issue was originally opened by @dkyrkou as hashicorp/terraform#12930. It was migrated here as part of the provider split. The original body of the issue is below.

---

Terraform Version

Terraform v0.9.1

Affected Resource(s)

Please list the resources as a list, for example:

• data external provider

Terraform Configuration Files

provider.tf

provider "aws" {
  profile                 = "saml"
  region                  = "${data.external.aws_auth.result["region"]}"
}

data "external" "aws_auth" {
     program = ["python", "${path.module}myscript.py"]
}

myscript.py

#!/usr/bin/python
import json
import sys
test_dict = {}
test_dict['region'] =  "eu-west-1"
json.dump(test_dict, sys.stdout)

Debug Output

When I run "terraform plan", I get the following error

Error running plan: 1 error(s) occurred:

* provider.aws: Not a valid region: 

When the python script is run by itself, it produces the following valid json:

{"region": "eu-west-1"}

Expected Behavior

The string "eu-west-1" should be extracted from the python-produced json and get assigned to "region" in provider.aws

Actual Behavior

When I run "terraform plan", nothing gets assigned to "region". As you can see, nothing exists next to "Not a valid region:" output.

Steps to Reproduce

1. terraform plan

Important Factoids

The truly weird thing is that if you modify provider.tf to the following

provider "aws" {
  profile = "saml"
  region = "${data.external.aws_auth.result["region"] == "" ? "eu-west-1" : "eu-west-2"}"
}

The assigned value is still nothing! How? As far as I know, the "?" operator is binary, which means that either "eu-west-1" or "eu-west-2" should get assigned to region, yet absolutely NOTHING is assigned!

Truly weird...

PS: Please no responses telling me that I should get rid of the script and assign the string directly to region, the actual production script reads from a configuration file, the provided script is just a small script that demonstrates the same problem.

[bflad]

This issue appears to be related to the upstream Terraform CLI feature request hashicorp/terraform#4149 or will potentially need to be filed against the Terraform AWS Provider issues if it still present. Terraform CLI may defer data source operations until after provider configuration occurs in certain scenarios. The receiving provider may be able to defer configuration, however that process is typically complex to handle properly across all resources/data sources of the provider, rather than upfront. That type of change can cause many other issues, beyond just reporting invalid configurations in a delayed manner for every individual resource/data source. The key here though is that the external provider cannot influence or change any of the behaviors of attempting to use data source values in provider configurations, so I'm going to close this issue.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.