External Data Source always forcing a replacement

[cyrus-mc]

Terraform Version

› terraform -v
Terraform v0.12.2
+ provider.aws v2.15.0
+ provider.external v1.2.0
+ provider.local v1.2.2
+ provider.null v2.1.2
+ provider.template v2.1.2

Your version of Terraform is out of date! The latest version
is 0.12.3. You can update by downloading from www.terraform.io/downloads.html

Terraform Configuration Files

data "external" "get_key" {
  program = [ "${path.module}/get_flux_key.sh" ]

  query = {
    kubeconfig = var.kubeconfig
  }

  /* ensure deployment is done before checking for key (script will also contain logic) */
  depends_on = [
    null_resource.deployment
  ]
}

data "template_file" "lambda_payload" {
  template = file("${path.module}/templates/lambda-payload.json.tmpl")

  vars = {
    domain      = var.domain                                                                                                                      environment = var.environment
    ssh_key     = data.external.get_key.result.key
  }
}

resource "local_file" "lambda_payload" {
  content = data.template_file.lambda_payload.rendered
  filename = "${path.module}/templates/rendered/lambda_payload.json"
}

The script get_flux_key.sh simply returns an SSH key (which is stable, calling that function over and over returns the same output.

When I run the above it always reports the following plan.

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
 <= read (data resources)

Terraform will perform the following actions:

  # module.flux.data.external.get_key will be read during apply
  # (config refers to values not yet known)
 <= data "external" "get_key"  {
      + id      = (known after apply)
      + program = [
          + ".terraform/modules/flux/get_flux_key.sh",
        ]
      + query   = {
          + "kubeconfig" = "./kubeconfig_kubernetes.test"
        }
      + result  = (known after apply)
    }

  # module.flux.data.template_file.lambda_payload will be read during apply
  # (config refers to values not yet known)
 <= data "template_file" "lambda_payload"  {
      + id       = (known after apply)
      + rendered = (known after apply)
      + template = jsonencode(
            {
              + domain      = "${domain}"
              + environment = "${environment}"
              + ssh_key     = "${ssh_key}"
            }
        )
      + vars     = {
          + "domain"      = "freight"
          + "environment" = "test"
          + "ssh_key"     = (known after apply)
        }
    }

  # module.flux.local_file.lambda_payload must be replaced
-/+ resource "local_file" "lambda_payload" {
      ~ content  = jsonencode(
            {
              - domain      = "freight"
              - environment = "test"
              - ssh_key     = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCwfdg+pS/zTP8mUo9TxXd5qEybTUevXGEZkTlXbVJOtoDWg8x9AMY25knlB+PfBH/LV3wgai56zFxH6RfwyYuftS52qauloj321ZdzKO89fhFAWvhIjjWFEzCSwAyzCEHlodpfCsLWPm2mY1/TCYhCMv7WoRInlp3JNPyopMgF/tJ8/WS1JaMbonwuDc/TKMzDjDfsKeDBnpXS9JVC5A5NJPulPRKGN2Fp0q8FgwzPcasxR8KC1O+GGHACKLkJP2VgegOZb8k+/RAYa6RxP8ieLy0ED16ygm9J7v7yde3uDRfVoZb5ClaGo4UP4Im1J+wp1FTBR9U4OPl0rfiaFjZX"
            }
        ) -> (known after apply) # forces replacement
        filename = ".terraform/modules/flux/templates/rendered/lambda_payload.json"
      ~ id       = "64f442851b2e378829fac94aeb25b0bd4d7f0776" -> (known after apply)
    }

Expected Behavior

If my understanding of the external data source is correct it should behave just like any other attribute. Only force a replacement when the value changes.

Actual Behavior

As shown above, resource local_file constantly wants to be replaced because the value for ssh_key isn't know until apply.

[cyrus-mc]

Seems my issue is related to #12326.

[apparentlymart]

Hi @cyrus-mc! Sorry for this surprising behavior.

Currently depends_on for data resources forces the read to always be deferred until apply time, meaning the results are always unknown during planning. depends_on for data resources is therefore useful only in some very specialized situations where that doesn't cause a problem, as discussed in the documentation.

In your case, the answer would be to represent the dependency on null_resource.deployment via an implicit dependency instead. For example, you could add a new key to query that the external script would ignore entirely but Terraform would still understand it as creating a dependency:

data "external" "get_key" {
  program = [ "${path.module}/get_flux_key.sh" ]

  query = {
    kubeconfig = var.kubeconfig

    # This is not used, but ensures that the read will be deferred until
    # after the deployment is done.
    deployment = null_resource.deployment.id
  }
}

The tracking issue for this limitation is #17034, which also includes a proposed architecture change that would address it. We're hoping to address it in a future major version of Terraform, but the changes required are quite invasive to Terraform's internals so we won't be able to tackle this in the near future without disturbing other work in progress.

Since we already have #17034 open, I'm going to close this issue just to consolidate the discussion over there. Thanks for reporting this, and I hope the above workaround will work for you in the meantime.

[cyrus-mc]

ends_on for data resources forces the read to always be deferred until apply time, meaning the results are always unknown during planning. depends_on for data resources is therefore useful only in some very specialized situations where that doesn't cause a problem, as discussed in the documentation.

@apparentlymart

Have a quick follow-up on this.

resource "local_file" "kubeconfig" {
  content  = var.kubeconfig
  filename = "${path.module}/kubeconfig"
}
data "external" "get_key" {
  program = [ "${path.module}/get_flux_key.sh" ]

  query = {
    kubeconfig       = local_file.kubeconfig.filename
    null_resource_id = null_resource.deployment.id
  }
}

So for get_flux_key to run resource local_file "kubeconfig" must have already been created. But I am finding that during the plan stage it tries to refresh the external data source (so run get_flux_key.sh) - which just ends up failing.

[cyrus-mc]

@apparentlymart

Assuming it runs because filename is known at plan time. What I really need is a way to run only when the file has actually been written to disk.

[faarshad]

@cyrus-mc - I am experiencing a very similar issue (terraform 0.12.3)

data "external" "flux_ssh_pub_key" {
  program = ["${path.module}/get_flux_pub_key.sh"]

  query = {
    namespace = local.namespace
  }

  depends_on = [
    helm_release.flux
  ]
}

flux script:

#!/bin/bash

set -e

eval "$(jq -r '@sh "NS=\(.namespace) FLUX=\(.flux)"')"

{
  SSH_RSA=$(fluxctl --k8s-fwd-ns="$NS" identity)
} &> /dev/null

echo {\"ssh-rsa\":\""$SSH_RSA"\"}

Error:

Error: command ".../get_flux_pub_key.sh" failed with no error message

  ... in data "external" "flux_ssh_pub_key":
 330: data "external" "flux_ssh_pub_key" {

I have also tried using the implicit dependency without the depends_on and had the same error.

Just wanted to share in case you had found a solution.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.