Terraform detects change when there is no change due to template_file

[thtran101]

I use Terraform to manage a serverless achitecture on AWS and after migrating to Terraform v0.12.2 from v011.x, I've noticed that there are "false" positive diffs detected when running plan/apply but the false positive change is not actually applied when the plan is approved. This problem revolves around the use of template file resources. It seems like there is a difference in how/when?? template files are rendered and evaluated against current state.

The following are my TF specs.

Terraform v0.12.2

• provider.aws v2.15.0
• provider.null v2.1.2
• provider.template v2.1.2

I've put together as concise an example for reproducing the behavior as possible. In my example below the template file is used for a resource policy, but I have this same problem occurring on a state function definitions using template files.

resource "aws_lambda_function" "test" {
  function_name = "test-delete-me"

  filename = "code-deployments/test.zip"
  handler  = "index.handler"
  runtime  = "nodejs10.x"

  // use any existing IAM role compatible w/ lambda to reproduce error
  role = aws_iam_role.lambda_basic_execution.arn

  publish = false
  timeout = 5

  environment {

    variables = {
      a_lambda_var = "x"
    }

  }

}

resource "aws_iam_role" "test_role" {
  name = "test-delete-me-role"

assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF

}

data "template_file" "test_policy" {
  /*
    Use any policy file, doesn't need to actually consume the variable below
  */
  template = file("policies/test_policy.tpl")

  vars = {
    my_var = aws_lambda_function.test.arn
  }
}

resource "aws_iam_role_policy" "test_role_policy" {
  name = "test-policy"
  role = aws_iam_role.test_role.id

  policy = data.template_file.test_policy.rendered

}

In the above configuration file there is:

• a lambda function which should use any existing IAM role. The function itself doesn't matter nor does the role.
• a test IAM role
• a template file for an IAM policy that is defined with a variable (attached below for convenience, actual content doesn't matter)
• an inline policy to be attached to the test IAM role

When the infrastructure has been deployed and is in a steady state with no diffs detected, deploy an update to the lambda by toggling the a_lambda_var to another value like "y".

Expected Behavior:
Only 1 change is detected with terraform apply/plan for the lambda function.

Actual Behavior:
2 changes are detected/predicted in the following order:
a) aws_iam_role_policy.test_role_policy will change with its single statement being dropped
b) lambda function changes due to variable value change

Actual Approved Plan Behavior:
Only 1 modification is made to the lambda function which contradicts the plan.

I didn't experience this problem in Terraform v0.11.x or earlier versions. I've used my config for over 6 months with countless deployments. This bug may be related to open issue 21545???

test_policy.txt

Let me know if you need me to attach a test lambda package, but absolutely any package will allow you to reproduce the problem.

[apparentlymart]

Hi @thtran101! Sorry for this confusing behavior, and thanks for reporting it.

You didn't share the actual plan output but I assume what you saw there was that the aws_iam_role_policy.test_role_policy.policy attribute was planned to change from a specific rendered string to (known after apply).

In that case, this seems like an AWS provider problem: it is responding to the change of environment variables by indicating that the arn attribute will take on a new, unknown value. That in turn causes Terraform to need to re-render the template and then, in turn, to potentially update the policy.

Because a Lambda function ARN is, in practice, a fixed string computed from the host account id and the function name, changing the environment variables doesn't actually change it, and so during apply Terraform notices that the policy document hasn't actually changed after all. This is one situation where Terraform can contradict itself: it might discover during apply that a planned step that was contingent on an unknown value isn't needed after all, and so will skip it. What isn't allowed is Terraform doing extra work that wasn't reflected in the plan.

That this was a new behavior after upgrading to 0.12.2 is surprising: what you are seeing is expected if the provider returns an unknown value for arn, so I assume that something has changed on the provider side and perhaps you upgraded the AWS provider at the same time as upgrading Terraform CLI and thus saw the behavior change at the same time?

I'm going to ask our bot to move this over to the AWS provider repository since I think any work to address it will need to be in that codebase. One potential solution is for the AWS provider to predict the final result of arn during planning: if there is a known value for function_name then the provider should be able to combine that with the current account id to compute the ARN itself and include it in the plan, thus allowing Terraform to see immediately that it isn't going to change. However, I'll leave the AWS provider team to investigate and see if a different approach might be better here.

[ghost]

This issue has been automatically migrated to hashicorp/terraform-provider-aws#9042 because it looks like an issue with that provider. If you believe this is not an issue with the provider, please reply to hashicorp/terraform-provider-aws#9042.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.