(aws-secretsmanager): Creating RotationSchedule when rotation lambda and secret are in different stack fails due to cyclic reference

[jacklin213]

Describe the bug

Currently when creating a RotationSchedule (either directly or using secret.addRotationSchedule) for a secret when the rotation lambda is in a different stack will cause cdk synth to fail with a cyclic reference error.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

We should be able to create a RotationSchedule for a secret even if the lambda function exists in another stack

Current Behavior

Currently when creating a RotationSchedule (either directly or using secret.addRotationSchedule) for a secret when the rotation lambda is in a different stack. cdk synth will fail with the below error

Error: 'SecretStack' depends on 'LambdaStack' ({SecretStack/TestSecret/RotationSchedule/Resource}.addDependency({LambdaStack/TestLambda/InvokeN0--a2GKfZP0JmDqDE--Vhhu6+A0TUv3NyNbk4YM+FKNc=})). Adding this dependency (LambdaStack -> SecretStack/TestSecret/Resource.Ref) would create a cyclic reference.
    at LambdaStack._addAssemblyDependency (C:\secret-rotation-lambda\node_modules\aws-cdk-lib\core\lib\stack.js:1:11960)
    at operateOnDependency (C:\secret-rotation-lambda\node_modules\aws-cdk-lib\core\lib\deps.js:1:1796)
    at addDependency (C:\secret-rotation-lambda\node_modules\aws-cdk-lib\core\lib\deps.js:1:532)
    at LambdaStack.addDependency (C:\secret-rotation-lambda\node_modules\aws-cdk-lib\core\lib\stack.js:1:9022)
    at resolveValue (C:\secret-rotation-lambda\node_modules\aws-cdk-lib\core\lib\private\refs.js:1:3840)
    at resolveReferences (C:\secret-rotation-lambda\node_modules\aws-cdk-lib\core\lib\private\refs.js:1:1473)
    at prepareApp (C:\secret-rotation-lambda\node_modules\aws-cdk-lib\core\lib\private\prepare-app.js:1:806)
    at synthesize (C:\secret-rotation-lambda\node_modules\aws-cdk-lib\core\lib\private\synthesis.js:1:1607)
    at App.synth (C:\secret-rotation-lambda\node_modules\aws-cdk-lib\core\lib\stage.js:1:2474)
    at process.<anonymous> (C:\secret-rotation-lambda\node_modules\aws-cdk-lib\core\lib\app.js:1:1767)

Unpacking the above, the RotationSchedule resource has a dependency on the generated AWS::Lambda::Permission which was added via #26512.

Even when using an aws-cdk version earlier than the change eg) v2.92.0 the request will fail with the below error

Error: 'LambdaStack' depends on 'SecretStack' (LambdaStack -> SecretStack/TestSecret/Resource.Ref). Adding this dependency (SecretStack -> LambdaStack/TestLambda/Resource.Arn) would create a cyclic reference.
    at SecretStack._addAssemblyDependency (C:\secret-rotation-lambda\node_modules\aws-cdk-lib\core\lib\stack.js:1:10330)
    at operateOnDependency (C:\secret-rotation-lambda\node_modules\aws-cdk-lib\core\lib\deps.js:1:1649)
    at addDependency (C:\secret-rotation-lambda\node_modules\aws-cdk-lib\core\lib\deps.js:1:321)
    at SecretStack.addDependency (C:\secret-rotation-lambda\node_modules\aws-cdk-lib\core\lib\stack.js:1:7442)
    at resolveValue (C:\secret-rotation-lambda\node_modules\aws-cdk-lib\core\lib\private\refs.js:1:3259)
    at synthesize (C:\secret-rotation-lambda\node_modules\aws-cdk-lib\core\lib\private\synthesis.js:1:922)
    at App.synth (C:\secret-rotation-lambda\node_modules\aws-cdk-lib\core\lib\stage.js:1:2052)
    at process.<anonymous> (C:\secret-rotation-lambda\node_modules\aws-cdk-lib\core\lib\app.js:1:1448)

I suspect this is because the lambda function needs to !Ref TestSecret for permissions and the RotationSchedule needs rotationLambdaArn: !GetAtt TestLambda.arn

Reproduction Steps

Repo: https://github.com/jacklin213/secret-rotation-lambda

app.ts

import * as cdk from 'aws-cdk-lib';
import { SecretStack } from '../lib/secret-stack';
import { LambdaStack } from '../lib/lambda-stack';

const app = new cdk.App();
const lambdaStack = new LambdaStack(app, 'LambdaStack', {
    env: { account: '000000000000', region: 'us-east-1' }
});
new SecretStack(app, 'SecretStack', {
    env: { account: '000000000000', region: 'us-east-1' },
    rotationLambda: lambdaStack.lambda
});

lambda-stack.ts

import * as cdk from 'aws-cdk-lib';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { Construct } from 'constructs';

export class LambdaStack extends cdk.Stack {
    readonly lambda: lambda.Function;
    constructor(scope: Construct, id: string, props?: cdk.StackProps) {
        super(scope, id, props);

        this.lambda = new lambda.Function(this, 'TestLambda', {
            runtime: lambda.Runtime.NODEJS_18_X,
            handler: 'index.handler',
            code: new lambda.InlineCode('console.log("works")')
        });
    }
}

secret-stack.ts

import * as cdk from 'aws-cdk-lib';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { Secret } from 'aws-cdk-lib/aws-secretsmanager';
import { Construct } from 'constructs';

interface SecretStackProps extends cdk.StackProps {
    rotationLambda: lambda.Function;
}

export class SecretStack extends cdk.Stack {
    readonly secret: Secret;
    constructor(scope: Construct, id: string, props: SecretStackProps) {
        super(scope, id, props);

        this.secret = new Secret(this, 'TestSecret');
        this.secret.addRotationSchedule('RotationSchedule', {
            automaticallyAfter: cdk.Duration.days(30),
            rotateImmediatelyOnUpdate: true,
            rotationLambda: props.rotationLambda
        });
    }
}

Then run cdk synth to see the issue

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.178.1 (build ae342cb)

Framework Version

No response

Node.js Version

v18.17.0

OS

Windows

Language

TypeScript

Language Version

5.6.3

Other information

No response

[pahud]

Can you share your use case why you have to separate RotationSchedule and rotation lambda in different stacks? I mean technically we should allow that but generally we recommend having them in the same stack whenever possible. Before we consider what's the best action we should do, can you share your use case and why it stops you from putting them in the same stack?

[xposionn]

Can you share your use case why you have to separate RotationSchedule and rotation lambda in different stacks? I mean technically we should allow that but generally we recommend having them in the same stack whenever possible. Before we consider what's the best action we should do, can you share your use case and why it stops you from putting them in the same stack?

It behaves the same if you want to create an ECS Cluster on stack A and ASG and Capacity provider in stack B

I'm following this issue in order to find solutions to my case

My reason of splitting both stacks that I was 1 stack for Cluster and 2 stacks for 2 ASGs, one for OneBox environment and one for Fleet env.

Today I have the same ASG for both OneBox and Fleet envs, during deployment and a change int the ASG will change the EC2s in both envs which is not my desire, I would like to have 2 ASG in distinct stacks and 1 shared Cluster

[jacklin213]

Can you share your use case why you have to separate RotationSchedule and rotation lambda in different stacks? I mean technically we should allow that but generally we recommend having them in the same stack whenever possible. Before we consider what's the best action we should do, can you share your use case and why it stops you from putting them in the same stack?

We group our resources by application, which includes secrets. However the credential rotation logic stays the same so instead of launching the lambda in each stack we just treat the lambda as a Singleton and want to use that to rotate secrets in StackA, StackB and StackC for example, while the lambda itself sits in StackA

I believe this would be a fairly common usecase