cloud-assembly-schema: MetadataEntryData type missing boolean | number #4742
Assignees
Labels
bug
This issue is a bug.
effort/medium
Medium work item – a couple days of effort
p1
potential-regression
Marking this issue as a potential regression to be checked by team member
Comments
matusfaro
added
bug
github-actions
bot
added
the
potential-regression
matusfaro
changed the title
|
@matusfaro Good morning. Thanks for opening the issue. You mentioned that the issue was likely introduced as part of the change introduced here. This change was made 5 months ago, but we haven't got any reports of regression until now. By definition of potential regression, the issue should likely be introduced in recently released version of CDK, which was working in last (or may be few versions, let's day 4-5 versions, in the past). Please share the CDK (CLI and Lib) version used. In order to reproduce the issue, please share the minimal self contained CDK code that reproduces the issue and demonstrates upgrading cloud schema from Thanks, |
ashishdhingra
added
p2
response-requested
|
Here's a simple self-contained repro in Groovy. Honestly I am not sure why I am the first one to hit this, it could be for a combination of reasons:
@Grab(group = 'software.amazon.awscdk', module = 'aws-cdk-lib', version = '2.173.4')
@Grab(group = 'com.fasterxml.jackson.core', module = 'jackson-databind', version = '2.15.2')
@Grab(group = 'software.amazon.awscdk', module = 'cdk-cloud-assembly-schema', version = '39.1.35')
import software.amazon.awscdk.App
import software.amazon.awscdk.Stack
import software.amazon.awscdk.services.s3.Bucket
import software.amazon.awscdk.services.s3.deployment.BucketDeployment
import software.amazon.awscdk.cloud_assembly_schema.AssemblyManifest
import software.amazon.awscdk.cloud_assembly_schema.Manifest
class MyStack extends Stack {
MyStack(scope, String id) {
super(scope, id)
Bucket destinationBucket = Bucket.Builder.create(this, "MyBucket")
.bucketName("my-deployment-bucket")
.build()
BucketDeployment.Builder.create(this, "MyBucketDeployment")
.sources(List.of())
.destinationBucket(destinationBucket)
.build()
}
}
def app = new App()
new MyStack(app, "MyStack")
def assembly = app.synth()
println "Created manifest.json:"
def manifestPath = "${assembly.directory}/manifest.json"
println manifestPath
// Read and print the manifest file content
def manifestFile = new File(manifestPath)
println manifestFile.text
// Parse the manifest
// THIS WILL FAIL
AssemblyManifest assemblyLoaded = Manifest.loadAssemblyManifest(manifestPath)
println assemblyLoadedOutputs: |
github-actions
bot
removed
the
response-requested
|
@matusfaro Good afternoon. Thanks for sharing the code. Ignoring details about language used (Groovy in your case), this appears to be JSII issue.
TypeScript examplelib/cdktest-stack-new.ts import * as cdk from 'aws-cdk-lib';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as s3deployment from 'aws-cdk-lib/aws-s3-deployment';
export class CdktestStackNew extends cdk.Stack {
constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const destinationBucket = new s3.Bucket(this, 'MyBucket', {
removalPolicy: cdk.RemovalPolicy.DESTROY
});
new s3deployment.BucketDeployment(this, 'MyBucketDeployment', {
sources: [],
destinationBucket: destinationBucket
});
}
}bin/cdktest.ts #!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { CdktestStackNew } from '../lib/cdktest-stack-new';
import * as fs from 'fs';
import * as cloudAssemblySchema from '@aws-cdk/cloud-assembly-schema';
const app = new cdk.App();
new CdktestStackNew(app, 'CdktestStackNew', {
env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION },
});
const assembly = app.synth();
const manifestPath = `${assembly.directory}/manifest.json`;
console.log(`manifestPath: ${manifestPath}`);
const manifestFile = fs.readFile(manifestPath, { encoding: 'utf-8'}, (err, data) => {
if (err) throw err;
console.log(data);
});
const assemblyLoaded = cloudAssemblySchema.Manifest.loadAssemblyManifest(manifestPath);
console.log(`assemblyLoaded.version: ${assemblyLoaded.version}`);package.json {
"name": "cdktest",
"version": "0.1.0",
"bin": {
"cdktest": "bin/cdktest.js"
},
"scripts": {
"build": "tsc",
"watch": "tsc -w",
"test": "jest",
"cdk": "cdk",
"integ-test": "integ-runner"
},
"devDependencies": {
"@eslint/js": "^9.14.0",
"@types/jest": "^29.5.12",
"@types/node": "20.14.9",
"aws-cdk": "2.174.0",
"eslint": "^9.14.0",
"globals": "^15.12.0",
"jest": "^29.7.0",
"ts-jest": "^29.1.5",
"ts-node": "^10.9.2",
"typescript": "~5.5.3",
"typescript-eslint": "^8.13.0"
},
"dependencies": {
"aws-cdk-lib": "2.174.0",
"@aws-cdk/cloud-assembly-schema": "39.1.38",
"constructs": "^10.0.0",
"source-map-support": "^0.5.21"
}
}Running
Java exampleCdktestjavaStack.java package com.myorg;
import software.constructs.Construct;
import software.amazon.awscdk.Stack;
import software.amazon.awscdk.StackProps;
import software.amazon.awscdk.services.s3.Bucket;
import software.amazon.awscdk.services.s3.deployment.BucketDeployment;
import java.util.List;
public class CdktestjavaStack extends Stack {
public CdktestjavaStack(final Construct scope, final String id) {
this(scope, id, null);
}
public CdktestjavaStack(final Construct scope, final String id, final StackProps props) {
super(scope, id, props);
Bucket destinationBucket = Bucket.Builder.create(this, "MyBucket")
.bucketName("my-deployment-bucket")
.build();
BucketDeployment.Builder.create(this, "MyBucketDeployment")
.sources(List.of())
.destinationBucket(destinationBucket)
.build();
}
}CdktestjavaApp.java package com.myorg;
import software.amazon.awscdk.App;
import software.amazon.awscdk.Environment;
import software.amazon.awscdk.StackProps;
import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;
import software.amazon.awscdk.cloudassembly.schema.AssemblyManifest;
import software.amazon.awscdk.cloudassembly.schema.Manifest;
import software.amazon.awscdk.cxapi.CloudAssembly;
public class CdktestjavaApp {
public static void main(final String[] args) {
App app = new App();
new CdktestjavaStack(app, "CdktestjavaStack", StackProps.builder()
.env(Environment.builder()
.account(System.getenv("CDK_DEFAULT_ACCOUNT"))
.region(System.getenv("CDK_DEFAULT_REGION"))
.build())
.build());
CloudAssembly assembly = app.synth();
System.out.println("Created manifest.json: ");
String manifestPath = assembly.getDirectory() + "/manifest.json";
System.out.println(manifestPath);
File manifestFile = new File(manifestPath);
try {
Scanner sc = new Scanner(manifestFile);
while (sc.hasNextLine())
System.out.println(sc.nextLine());
} catch (FileNotFoundException e) {}
AssemblyManifest assemblyLoaded = Manifest.loadAssemblyManifest(manifestPath);
System.out.println(assemblyLoaded.getVersion());
}
}pom.xml <?xml version="1.0" encoding="UTF-8"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"
xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<modelVersion>4.0.0</modelVersion>
<groupId>com.myorg</groupId>
<artifactId>cdktestjava</artifactId>
<version>0.1</version>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<cdk.version>2.174.0</cdk.version>
<constructs.version>[10.0.0,11.0.0)</constructs.version>
<junit.version>5.7.1</junit.version>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.11.0</version>
<configuration>
<release>11</release>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>3.1.0</version>
<configuration>
<mainClass>com.myorg.CdktestjavaApp</mainClass>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<!-- AWS Cloud Development Kit -->
<dependency>
<groupId>software.amazon.awscdk</groupId>
<artifactId>cdk-cloud-assembly-schema</artifactId>
<version>39.1.38</version>
</dependency>
<dependency>
<groupId>software.amazon.awscdk</groupId>
<artifactId>aws-cdk-lib</artifactId>
<version>${cdk.version}</version>
</dependency>
<dependency>
<groupId>software.constructs</groupId>
<artifactId>constructs</artifactId>
<version>${constructs.version}</version>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
Running cdklabs/cloud-assembly-schema was migrated from main CDK repo as separate repo, starting from version If we use Transferring to JSII repo. Thanks, |
ashishdhingra
added
p1
effort/medium
|
@matusfaro thanks for reporting. I was able to reproduce and it seems like the cause is indeed the change you mentioned, unrelated to the That is, a meer bump of |
|
@iliapolo Yes that's right, it seems the new schema introduces values that the type doesn't have. So likely the type just needs to be updated. |
mergify bot
pushed a commit
to cdklabs/cloud-assembly-schema
that referenced
this issue
Jan 8, 2025
In aws/aws-cdk#31041, the CDK started emitting booleans and numbers as metadata values. However, since these types are not officially declared in the schema, jsii runtime type checking prevents loading them: ```console [ERROR] ├── 🛑 Failing value is a boolean [ERROR] │ true [ERROR] ╰── 🔍 Failure reason(s): [ERROR] ├─ [as string] Value is not a string [ERROR] ├─ [as array<aws-cdk-lib.cloud_assembly_schema.Tag>] Value is not an array [ERROR] ├─ [as aws-cdk-lib.cloud_assembly_schema.FileAssetMetadataEntry] Value is not an object [ERROR] ╰─ [as aws-cdk-lib.cloud_assembly_schema.ContainerImageAssetMetadataEntry] Value is not an object ``` Fixes aws/jsii#4742 > Note that releasing this will immediately resolve the above issue, no cdk release needed. It does bump the schema version, which we can update at will. ## FAQ ### If the types are not allowed in the schema, how can a manifest with them be written? Actually, these types **are** allowed in the schema, in the json schema that is: https://github.com/cdklabs/cloud-assembly-schema/blob/f5ae537437531364646e7515e5286e21583c8421/schema/cloud-assembly.schema.json#L131-L133 This free form data is added to the json schema in order to support calls to `node.addMetadata(key: string, value: any)`, as defined in the `constructs` API. However, this type was intentionally left out of the type declaration: https://github.com/cdklabs/cloud-assembly-schema/blob/f5ae537437531364646e7515e5286e21583c8421/projenrc/update-schema.ts#L79-L89 I'm fairly certain the comment is not applicable anymore, because we don't run jsii compatibility checks on the schema (we just bump it on every change). In practice, I think we can probably get rid of this patching and just type metadata entry values as `any` - but that seems like a bigger change I don't want to go into right now. ### If such manifests cannot be loaded, how does synth work? Because during synth, loading the assembly using the `Manifest` class happens solely inside the javascript space, it doesn't callback to java land. This problem presents itself **only when** directly using the java `Manifest` class and invoking the `loadAssemblyManifest` function on a manifest produced by `app.synth()`.
|
@iliapolo @ashishdhingra Thank you so much for the quick turnaround! Really appreciate it |
|
@matusfaro this should be resolved now. BTW I noticed you're using this import: import software.amazon.awscdk.cloud_assembly_schema.ManifestIts ok, but by using this you're actually picking up the import software.amazon.awscdk.cloudassembly.schema.Manifest;With the following dependency (which you already are): <dependency>
<groupId>software.amazon.awscdk</groupId>
<artifactId>cdk-cloud-assembly-schema</artifactId>
<version>39.1.41</version>
</dependency>You'll also need to bump your CDK cli to the latest version because you're picking up new schema versions. Note that a fix for the |
|
This issue is now closed. Comments on closed issues are hard for our team to see. |
matusfaro
added a commit
to datasprayio/aws-cdk-4j
that referenced
this issue
Jan 8, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Bumping to latest dependencies and going from cloud schema 36.0.0 to 38.0.1, I get the following error:
Notice that the
datatype is expected to bestring | aws-cdk-lib.cloud_assembly_schema.FileAssetMetadataEntry | aws-cdk-lib.cloud_assembly_schema.ContainerImageAssetMetadataEntry | array<aws-cdk-lib.cloud_assembly_schema.Tag> | undefined.This is because as part of the change introduced here:
The value of the
datafield may now be a number and boolean. But the type defined in cloud schemaMetadataEntryTypeis not updated with these types hereNotice these are the types from the error message.
Below is a snippet of the generated
manifest.jsonwith this issue:Note that
aws:cdk:is-custom-resource-handler-singletonmatches theCUSTOM_RESOURCE_SINGLETONabove andaws:cdk:is-custom-resource-handler-runtime-familymatches theCUSTOM_RESOURCE_RUNTIME_FAMILYabove.Regression Issue
Last Known Working CDK Version
CDK 2.134.0 Cloud Assembly 36.0.6
Expected Behavior
Shouldn't fail. Expected the generated manifest.json to be read properly.
Current Behavior
Fails with error.
Reproduction Steps
It's difficult to create a self-contained snippet for reproduction. It's unclear to me what conditions are met for the metadata entries to be generated.
Possible Solution
Add missing types to
MetadataEntryDataAdditional Information/Context
No response
CDK CLI Version
N/A
Framework Version
No response
Node.js Version
v20.11.0
OS
osx
Language
Java
Language Version
21.0.1-graalce
Other information
No response
The text was updated successfully, but these errors were encountered: