CDK Pipelines : assigning built code to a lambda

[FenryrMKIII]

General Issue

I have a working CDK pipeline which builds my code for my lambda function. I would like now to assign this S3 bucket holding my code to a Lambda function. But I can't find the proper way to perform this transfer of information from my CodeBuildStep to my Lambda stack

The Question

I have the following pipeline stack which works.

import * as cdk from 'aws-cdk-lib';
import * as codepipeline_actions from 'aws-cdk-lib/aws-codepipeline-actions';
import * as codepipeline from 'aws-cdk-lib/aws-codepipeline'
import * as codebuild from 'aws-cdk-lib/aws-codebuild'


import { Construct } from 'constructs';
import { CodePipeline, CodePipelineSource, ShellStep, CodeBuildStep } from 'aws-cdk-lib/pipelines';

export class CdkPipelineStack extends cdk.Stack {
    constructor(scope: Construct, id: string, props?: cdk.StackProps) {
        super(scope, id, props);

        // parameters
        const ConnectionArn = 'arn:aws:codestar-connections:region:account:connection/xxx-yyyy'


        // Source
        const cdkCodeStarConnection = CodePipelineSource.connection('owner/cdk-repo', 'master', {
            connectionArn: ConnectionArn
        })

        const lambdaCodeStarConnection = CodePipelineSource.connection('owner/lambda-repo', 'master', {
            connectionArn: ConnectionArn
        })

        // Building Lambda
        const lambdaBuildStep = new CodeBuildStep('BuildLambda', {
            buildEnvironment: {
                buildImage: codebuild.LinuxBuildImage.AMAZON_LINUX_2_3
            },
            input: lambdaCodeStarConnection,
            commands: ['variouscommands'],
            partialBuildSpec: codebuild.BuildSpec.fromObject({
                phases: {
                    install: {
                        "runtime-versions": {
                            python: "3.9"
                        },
                        "commands": ["python --version"]
                    }
                }
            })
        })

        const pipeline = new CodePipeline(this, 'Pipeline', {
            pipelineName: 'Pipeline',
            synth: new ShellStep('Synth', {
                input: cdkCodeStarConnection,
                commands: ['npm ci', 'npm run build', 'npx cdk synth', 'npx cdk ls'],
                additionalInputs: { 'lambda': lambdaBuildStep.addOutputDirectory('lambda') }
            })
        });
    }
}

I have a second stack featuring a Lambda function.

import { LambdaMonolith } from './lambda-construct';
import { Stage, StageProps } from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as cdk from 'aws-cdk-lib';

export class LambdaStack extends cdk.Stack {
    constructor(scope: cdk.App, id: string, props?: StageProps) {
        super(scope, id, props);

        const s3Code = new lambda.S3Code(bucket, 'key');

        const LambdaFunction = new lambda.Function(this, 'MyFunction', {
            code: s3Code ,
            runtime: lambda.Runtime.PYTHON_3_8,
            handler: 'lambda.handler',
          });
    }
}

I would like the s3Code to point to the S3 bucket created by the CodeBuildStep but I can't figure out how to properly transfer the information between the pipeline and the Lambda stack. Any help would be greatly appreciated.

CDK CLI Version

2.8.0

Framework Version

No response

Node.js Version

No response

OS

Windows

Language

Typescript

Language Version

4.5.4

Other information

No response

[bailout00]

Were you ever able to find the answer to this? I've seen some scary manual ways to do this but nothing built-in yet. I'm at 2.42.x

[nastycoder]

I think I'm going to have to rethink my approach here considering it's been nearly a year and still no traction on this question. But for giggles here's my example:

class MyPipeline extends Stack {
    constructor(scope: Construct, id: string, props?: StackProps) {
        super(scope, id, props);

        const artifactBucket = new Bucket(this, `${SERVICE_NAME}Artifacts`, {
            bucketName: `${SERVICE_NAME}Artifacts`.toLowerCase(),
            blockPublicAccess: BlockPublicAccess.BLOCK_ALL,
            encryption: BucketEncryption.S3_MANAGED,
            enforceSSL: true,
            versioned: true,
            removalPolicy: RemovalPolicy.RETAIN,
        });

        const lambdaBuild = this.getLambdaBuild();
        const lambdaFileSet = lambdaBuild.addOutputDirectory('dist');

        const pipeline = new CodePipeline(this, SERVICE_NAME, {
            artifactBucket: artifactBucket,
            dockerEnabledForSelfMutation: true,
            crossAccountKeys: true,
            pipelineName: SERVICE_NAME,
            synth: new ShellStep('Synth', {
                input: CodePipelineSource.connection(`${GITHUB_ORG}/${SERVICE_NAME}CDK`, 'main', {
                    connectionArn: GITHUB_CONNECTION
                }),
                additionalInputs: {
                    "lambda": lambdaBuild
                },
                commands: ['npm ci','npm run build','npx cdk synth'],
                env: {/* This is environment vars for the build environment */}
            })
        });

        const stage = pipeline.addStage(new LambdaStage(this, `${SERVICE_NAME}Lambda`, lambdaFileSet))
    }

    getLambdaBuild() {
        return new CodeBuildStep(`${SERVICE_NAME}Lambda`, {
            input: CodePipelineSource.connection(`${GITHUB_ORG}/${SERVICE_NAME}Lambda`, 'main', {
                connectionArn: GITHUB_CONNECTION
            }),
            commands: [],
            buildEnvironment: {
                privileged: true,
            },
            partialBuildSpec: BuildSpec.fromObject({
                phases: {
                    build: {
                        commands: [
                            "docker build -t lambda .",
                            "docker run --rm -v `pwd`:/project lambda bash -c 'cd /project; gradle build'"
                        ]
                    },
                    post_build: {
                        commands: [
                            "mv ./app/build/dist ./",
                            "mv ./app/build/reports ./dist/"
                        ]
                    }
                }
            })
        });
    }
}

class LambdaStage extends Stage {
    constructor(scope: Construct, id: string, files: FileSet) {
        super(scope, id);

        const stack = new LambdaStack(this, `${id}Stack`, files);
    }
}

class LambdaStack extends Stack {
    constructor(scope: Construct, id: string, files: FileSet) {
        super(scope, id);

        const fn = new Function(this, 'Example', {
            code: new AssetCode(files.get('package.zip')),
            runtime: Runtime.JAVA_17,
            handler: 'com.example.Fancy::handeRequest'
        })
    }
}

In short, the CodeBuildStep outputs a directory structure that include a package.zip file at it's root (I've verified that that part is in fact working as expected) but I can't seem to find a way to get that zip file out of the produced fileset to pass it into my lambda creation.

One this is working I'd like to also put a approval step to ensure the test reports (also produced by the CodeBuildStep) are all passing. But that's a different problem that should be easy to find a solution too once I get this part working.

[abend-arg]

Look, I have tried a lot of times to have the CDK and services code in different packages without relying on hacky approaches without success. If there is a way of achieving that in the right way that is not correctly covered in the documentation. Save yourself time and effort by having everything in a mono repo, that was the best approach for me.

[nastycoder]

I understand the multi-repo approach isn't ideal but it adds an extra layer of separation between modules that gives me warm and fuzzies.

The multi-repo issues aside, I would still have the same issue described above. I can't get an artifact produced by a build step into the deployment stage.

[nastycoder]

For anyone else that stumbles into this thread, I'd like share my workaround. It's not an ideal solution, but I think it'll work.

export class MyPipeline extends Stack {
    constructor(scope: Construct, id: string, props?: StackProps) {
        super(scope, id, props);

        const pipeline = new CodePipeline(this, SERVICE_NAME, {
            dockerEnabledForSelfMutation: true,
            crossAccountKeys: true,
            pipelineName: SERVICE_NAME,
            synth: new ShellStep('Synth', {
                input: CodePipelineSource.connection(`${GITHUB_ORG}/${SERVICE_NAME}CDK`, 'main', {
                    connectionArn: GITHUB_CONNECTION
                }),
                commands: ['npm ci','npm run build','npx cdk synth'],
                env: {/* This is environment vars for the build environment */}
            })
        });

        const bootstrapStage = pipeline.addStage(new BootstrapStage(this, `${SERVICE_NAME}BootStrap`));

        /**
         * In order for this part to work; you have to deploy once without it, otherwise the
         * CfnOutput's do not exist.
         */
        const publishRole = <Role>Role.fromRoleArn(this, 'PublishRole', Fn.importValue('publishRoleArn'));
        const codeBucket = <Bucket>Bucket.fromBucketArn(this, 'CodeBucket', Fn.importValue('codeBucketArn'));
        bootstrapStage.addPost(this.buildAndPublishLambda(
            publishRole, codeBucket
        ));

        const lambdaStage = pipeline.addStage(
            new LambdaStage(this, `${SERVICE_NAME}Lambda`, { codeBucket: codeBucket })
        );
    }

    buildAndPublishLambda(role: Role, bucket: Bucket) {
        return new CodeBuildStep(`${SERVICE_NAME}Lambda`, {
            input: CodePipelineSource.connection(`${GITHUB_ORG}/${SERVICE_NAME}Lambda`, 'main', {
                connectionArn: GITHUB_CONNECTION
            }),
            commands: [],
            buildEnvironment: {
                privileged: true,
            },
            role: role,
            partialBuildSpec: BuildSpec.fromObject({
                phases: {
                    build: {
                        commands: [
                            "docker build -t lambda .",
                            "docker run --rm -v `pwd`:/project lambda bash -c 'cd /project; gradle build'"
                        ]
                    },
                    post_build: {
                        commands: [
                            "mv ./app/build/dist ./",
                            "mv ./app/build/reports ./dist/",
                            `aws s3 cp ./dist s3://${bucket.bucketName} --recursive`
                        ]
                    }
                }
            }),
            primaryOutputDirectory: 'dist'
        });
    }
}

class BootstrapStage extends Stage {
    stack: S3Stack;
    constructor(scope: Construct, id: string) {
        super(scope, id);

        this.stack = new S3Stack(this, `${id}Stack`);
    }
}

class S3Stack extends Stack {
    constructor(scope: Construct, id: string) {
        super(scope, id);

        const bucket = new Bucket(this, 'CodeStorage', {
            bucketName: `${id}-code-storage`.toLowerCase()
        });

        const role = new Role(this, 'PublishCodeRole', {
            assumedBy: new ServicePrincipal('codebuild.amazonaws.com')
        });

        bucket.grantPut(role);

        new CfnOutput(this, 'codeBucketArn', {
            value: bucket.bucketArn,
            exportName: 'codeBucketArn'
        });
        new CfnOutput(this, 'publishRoleArn', {
            value: role.roleArn,
            exportName: 'publishRoleArn'
        });
    }
}

interface LambdaStackProps {
    codeBucket: Bucket;
}

class LambdaStage extends Stage {
    constructor(scope: Construct, id: string, props: LambdaStackProps) {
        super(scope, id);

        const stack = new LambdaStack(this, `${id}Stack`, props);
    }
}

class LambdaStack extends Stack {
    constructor(scope: Construct, id: string, props: LambdaStackProps) {
        super(scope, id);

        const fn = new Function(this, 'Example', {
            functionName: 'Example',
            code: new S3Code(props.codeBucket, 'package.zip'),
            runtime: Runtime.JAVA_17,
            handler: 'com.example.Example::handleRequest'
        });
    }
}

[RickahTech]

Does your lambda trigger at that lambda stage

[nastycoder]

I don't understand the question.

The bootstrap stage builds an S3 bucket that's passed into the post stage task which builds the lambda and published the artifact to the previously created S3 bucket. The lambda stack then builds the lambda from the artifact in S3.

Like I said, it's not ideal but it worked for that project.