gem5 Resources: How do we ensure data integrity?

[BobbyRBruce]

We require tooling to update our gem5 Resources data, but there are many things to consider when doing so, most which regard maintaining the integrity of the data. Here I outline my thoughts on this.

Terminology

To start, I wish to clarify the terminology I'll be using in explaining using the following terminology.

• gem5 Resource: An object which can be constructed within the gem5 Python configuration script and passed into the simulation description, typically to declare the setting up of a workload for a simulation to run. These can be constructed manually or by obtaining data via the gem5 Resource data sources. For example a DiskImageResource object in the gem5 standard library contains a link/path to a Disk image and data regarding it (source, description, partitioning information, etc.). The set_disk_image_workload function in gem5 accepts a DiskImageResource in order to setup the disk image on for a simulation. The gem5 stdlib accepts manual construction of this, or may do so automatically by querying the gem5 Resources data sources for a resource, in which case the gem5 Resource is constructed based on the outcome of that query.
• gem5 Resources data source: What the gem5 stdlib reads to construct these gem5 Resource. Each entry in the data source corresponds to a gem5 Resource which may be used. Typically the gem5 stdlib will be passed the ID and version of a gem5 Resource, the data source will be queried for the entry with that ID and version. The entry will be returned is then constructed into a gem5 Resource. For now the data sources are either a JSON file or a Mongo database. By default it is the gem5 project's Mongo Database.
• gem5 Resources Schema: The Schema in which the gem5 Resource data sources conform to.

Things to know and current caveats

• The key of each entry in the gem5 Resource data sources is the ID and Version of a Resource.
• Entries with the same ID may have multiple versions.
• Workloads are special gem5 Resource's which define everything needed to run a specific workload in gem5. As such they may contain references to other Resources. E.g., a Workload may contain a reference to another DiskImageResource, KernelResource, and specify a the readfile_contents to pass to gem5 once the Kernel and Disk Image have been added to the system. Note: At the time of writing the Workloads are not technically a type of resource. This is highlighted in Issue Add workload to resource specialization of resources.py and deprecate workload.py #193. The remainder of this discussion will assume Workloads are a specialization of a Resource.
• At present Workloads' references to Resources do not specify the version of those resources. This is highlighted in Issue Workloads lacks specifying constituent Resource version: suggest change to gem5 Resources JSON schema. #192. I will continue this discussion assuming each Resource specified in a Workload is unique (e.g., specifies both an ID and a version).
• There is a new Resource type under development called Suites. Suites are collections of Workloads. These are useful for defining benchmark suites, for example. I will continue this discussion assuming these have already been implemented.
• Each resource has a compatible-versions field which is a list highlighting which versions of gem5 the resource is compatible with.
• The gem5 stdlib only requires the passing of an ID to obtain a Resource. gem5 will always get, unless explicitly stated otherwise, the latest version of a resource in which it is compatible. E.g., if v1 of a resource is compatible with v23.0 of gem5 and v2 of the same resource is compatible with v23.1, then v23.0 of gem5 would obtain v1 of that resource and a v23.1 of gem5 would obtain v2.

Goals

In maintaining this data over time we have the following goals:

1. If a gem5 Resource X is to be updated then a new entry of that resource is created in the data source. It new version will be Y+1 where Y was the value of the current latest version of X
2. When a new version of a resource is created the creator must explicitly state its compatibility with different versions of gem5 via the compatible-versions field.
3. If gem5 Resource X of version Y is updated (new entry with version Y+1), any Workloads referencing X of version Y must too be updated to include this new version (again a new entry is created with the same ID but a "+1ed" version) with this new Resource. This ensures that A) we preserve the older workload, and B) update the workload to contain the newer workload resource.
4. When a Workload is updated a Suite containing a Workload should also be updated in the same manner. E.g., we cannot update Resource X Version Y to Y+1 if Y+2 already exists.
   5.A Workload/Suite's compatible-versions field must be a subset of the intersection of all compatible-versions fields of it's constituent resources.

Pain Points

1. Handling chains of updates: If we update a Resource we need tooling in placed to automatically go through the data sources and update Workloads that contain it. Updated workloads need the Suites that contain them updated.
2. Schema Changes: It will become necessary over time to change the schema. This puts a hard constraint on a Resource's compatibility with different versions of gem5. If a newer version of gem5 uses a different schema then all the resources must be updated to this new version and have this new schema. The compatibility of these resources will therefore only be of that compatible with those versions of gem5 with compatibility to these schemas.
3. Compatible versions: What does it mean for a resource to be compatible with a version of gem5? It means that A) the data in the source is of a compatible schema for that version of gem5; and B) the Resource is itself compatible with gem5. When handling 1. we must be careful to update the Resources in the chain to have the correct compatible-versions field. A suite/workload with compatibility to Version Z of gem5 must not contain any resources which do not have compatibility to version Z/
4. Dead Workloads/Suites: There may be a case where a Workload/Suite is no longer compatible with any version of gem5. Should these continue to be updated in accordance to 1? The argument in favor of this is that we may be updating a lot of resources to a new version, the suite may have to go through many versions to reach a state where all its constituent resources have been updated before itself having a compatible version.

Open Questions

1. In the case of Suites/Workloads we could enforce that the compatible_versions of the Suite/Workloads is always the intersection of all the constituent Resources but what if the Schema of the Suite/Workload chances? That means it can only be compatible with gem5-verisons compatible with that schema even if the intersection of the constituent resources suggests more. How to handle this in an automated fashion?

[powerjg]

Not sure if I agree with the following:

5.A Workload/Suite's compatible-versions field must be a subset of the intersection of all compatible-versions fields of it's constituent resources.

I think that we should just have compatible-versions in one place and use the "recursion" to build the workload/suite compatible versions. I.e., can we just have compatible-versions for the disk images and not the workloads that use them?

---

I think that compatibility should be more about schema compatibility than tested with gem5. In theory, gem5 should be perfectly forward compatible in terms of ISA/devices/etc. and the only time it's not compatible is because a schema changes. This could be the json schema, the checkpoint "schema," or the stdlib api "schema."

I think that tested-with-versions instead of compatible-versions is an orthogonal (though also important) topic.

I'm not sure I totally understand the open question.

[BobbyRBruce]

We are inevitably going to make versions of resources that are incompatible with older versions of gem5. It will happen, and not just because of schema changes. For example, if we update the m5 libs Exit Events to return an object rather than just a string, we'll create new disk images which will not work as intended with older versions of gem5.

Given a workload depends on resources to run, we thought it best that a Workload's compatible-version is the intersection of its constituent resources. This makes sense. If a workload contains two resources, A and B, with A being compatible with Version 1 and 2, and B being compatible with just 2, then that workload should only have a stated compatibility of 2. It will simply not work with 1. This is nice because it's really easy to automate this. As we update resources we can quickly update Workloads and then Suites using this rule.

The open question regards, what if the Workload in this example has been updated to a new schema only compatible with version 3? What do we do here? Our automation doesn't work as before as it's no longer the intersection of the constituents. We'd have to manually fix it somehow. The open question is really: we need this to be automated as it's too much of a headache otherwise, but there comes a point we can't and need to manually fix it. How do we do both?

[powerjg]

Just to be sure that I understand, your general rule is that as gem5 is updated we will assume all Resources are "forwards compatible" and update them with the new compatible-versions that include the new version of gem5? Then, for new Resources, we'll assume they are only compatible with the current version and not backwards compatible (unless told otherwise)?

We are inevitably going to make versions of resources that are incompatible with older versions of gem5. It will happen, and not just because of schema changes. For example, if we update the m5 libs Exit Events to return an object rather than just a string, we'll create new disk images which will not work as intended with older versions of gem5.

Makes sense, though I hope this is very rare.

Given a workload depends on resources to run, we thought it best that a Workload's compatible-version is the intersection of its constituent resources. This makes sense. If a workload contains two resources, A and B, with A being compatible with Version 1 and 2, and B being compatible with just 2, then that workload should only have a stated compatibility of 2. It will simply not work with 1. This is nice because it's really easy to automate this. As we update resources we can quickly update Workloads and then Suites using this rule.

I think this makes sense as a rule to update the workload's compatible-versions field.

The open question regards, what if the Workload in this example has been updated to a new schema only compatible with version 3? What do we do here? Our automation doesn't work as before as it's no longer the intersection of the constituents. We'd have to manually fix it somehow. The open question is really: we need this to be automated as it's too much of a headache otherwise, but there comes a point we can't and need to manually fix it. How do we do both?

I think the Workload will have to change with the Resource. I don't see any way to do this except for manual inspection. For instance, if the disk image is updated, whoever updates the disk image should say "the workload now needs to execute this new command" or "the workload still works and doesn't need to change."

[BobbyRBruce]

I don't think I understand your general rule or where you got it from. I don't think it feeds into anything we discussed here. Resources are just compatible with the versions of gem5 they work with. If we release a new version of gem5 we may, as a quick hack, ensure all those compatible with the previous version are compatible now and add the new release as a compatible version, but we should probably run some tests where we can. This whole assumption ignores schema changes. If a reources's schema i no longer compatible with the new release, it's not compatible.

All the rest you seem to understand and be in agreement. On your last point about workloads: Yes, I think we need to update Workloads. In response to @Harshil2107 's idea I've proposed something else which might make things easier.

[BobbyRBruce]

[this is a writeup of a document hand-written by @Harshil2107 on the schema. His work is quoted, my response to his document is not]

Schema Versioning

Assumption: Schema versioning like gem5 will be floats: e.g. 23.1, 23.5, 25.0, etc.

No, they are just strings, we cannot cast them as floats. For example, the current release of gem5 is v23.0.1.0. That's not a float. This doesn't' change your coming points so it's a bit of a moot point. They are still sortable as strings though.

Schema versions will follow min-version to check compatibility. E.g.: Let us say we have a schema with 2 types of resources, let's call them 'binary' and 'workload'.

1. On gem5 version 1.0 a schema was made. (Total schemas: "schema_1.0.json"). All resources in the DB are compatible with '1.0' so "gem5_versions: ["1.0"] is the field in all resources.

This assumption isn't necessarily true. We may have resources that are compatible only with the upcoming release. This may be a nit-pick but just something to remember.

2. On gem5 version 1.1 the schema was the same. (Total schemas: "schema_1.0.json"). As no change in schema was made all resources should be compatible with "1.1" so gem5_versions will have ['1.0', '1.1']. Note: maybe make them semi-manual where users can define what categories have changed and are not compatible with '1.0'. Assumption: as resources are specialized any change will affect all resources or the same category.

No, we can't say this at all. If we have made a resource that is only compatible with version 1.1 of gem5, even if it has the same schema as that used in 1.0, we can't say it is compatible with 1.0. Being incompatible with the schema isn't the only way a resource may not work with a given version of gem5. For example, if we introduce a new RISC-V ISA instruction in gem5 v1.1, and we create a binary resource which uses it, that binary will not run in gem5 v1.0.

Your assumption is true, only the categories that have changed need to worry about compatibility problems in the new version. It might be hard to automate though due to resource specialization. E.g. if the schema for a File resource changes than that also impacts the schema for the Binary resource. This assumption may lead to a design that's more efficient in some ways but too much of a pain to engineer and maintain. If we update the schema, why not just assume anything that touched the old schema needs updated and create a new versions? Does it make much of a difference? We're just wasting database space but it's cheap.

3. In gem5 version '1.2' the schema for "Workload" changed. (Total schemas: 'schema_1.0.json', 'schema_1.2.json'). We therefore add '1.2' to gem5_version of all resources categories exception "workload unless specified otherwise.

You're not wrong here but we may want/need a way to automatically update all the resources to this new schema version (e.g., create a new version of each workload which conforms to the new schema). We don't want to lose, say, every workload for the next version of gem5 because we update it's schema. Don't prioritize this problem, it's definitely something we can work on later, but we will likely need to do this.

4. We validate the data against the schemas. We have 2 schemas and 3 gem5 versions (gem5 versions: '1.0', '1.1', '1.2'' schema: 'schema_1.0.json', 'schema_1.2.json"). To validate a resource we get elements from 'gem5_versions' field and then use the respective schema to validated it.
   1. gem5 versions: ['1.0', '1.1', '1.2'].
   2. For '1.0' we select schema_1.0.json and use it to check that data.
   3. For '1.1' there is no specific schema, this implies that that the schema has not changed since the last update, we get the last changed schema 'schema_1.0.json'.
   4. The '1.2' will use schema_1.2.json to check.
   5. If the resource is valid for all versions then only it will be considered a valid version. If not then an error will be thrown.

Wouldn't it be easier to just create a schema for each major release and avoid this logic? It's not saving us very much by being clever for gem5 versions that don't change the schema. An alternative approach to all this is to this is just to have a CSV file somewhere accessible (like the resources.gem5.org website) with a format like:

v1.1,https://resources.gem5.org/public/schema_1.json
v1.2,https://resources.gem5.org/public/schema_1.json
v1.3,https://resources.gem5.org/public/schema_A.json

In this approach we don't need to worry about what the file is called and match on it's name, we just look up our current version in the CSV, download the schema, then use it. It also makes it quick to get the most up-to-date schema. You just sort the CSV on the first column and select the first entry.

There is an assumption in this work that minor and hotfix releases don't exist. I'm fine with not changing the schema between major releases of gem5 but it should be explicitly noted if so.

Another approach to this

I think we need to step back and ask what it means for a resource to be compatible with a given version of gem5. It means two things:

1. That the resource itself is compatible with gem5. That is the disk image, binary, checkpoint, etc., if loaded into gem5 will work. We 100% will have cases where binary A, version 1, compatible with gem5 version X, will be updated to version 2 and it will only work with gem5 Version X+1. So version 1 and version 2 have completely different gem5 compatibilities
2. The data entry for the resource conforms to a schema that version of gem5 can process. Resource binary B version 5 may not change at all between gem5 version X and X+1 but X+1 may enforce a difference schema that works in a manner which version 5's entry in the database is no longer compatible. Version 6 would be created in accordance to this new schema and version 6 would only be compatible with version X+1.

Given this wouldn't it be good to have the following fields:

1. schema: <X> : A field specifying which schema this resource conforms too. This could be a url to the schema itself or perhaps just a version of the schema to go find (I can't think which is better...)
2. resource_compatibility: List[<gem5_versions>]: A field specifying the compatibility of the resource itself (ignore the data here, i'm talking the actual resource. Like the binary, or the simpoint). So a binary like "hello world" is probably gonna continue to be compatible with ALL versions of gem5, despite how it's schema changes.
3. gem5_version: List[<gem5_versions>]: Now, this is the clever part. This is an automatically generated field it takes the intersection of the resource_compatibility values and the gem5 versions compatible with the schema outlined in the schema field. The schema compatibility could parse the CSV I mentioned earlier, or we could have the compatible gem5 versions of a schema imbedded in the schema file itself.

We could easily write scripts to automatically go through the database to update the gem5_versions field in accordance to the above. There would be a special case for workloads. They would work by having their resource_compatibility field automatically generated as the intersection of all the gem5_version fields of the constituent resources. Then from there, like all other resources, the gem5_version would be the intersection of this automatically generated resource_compatibility field and the workload's schema compatibility. We'd do a very similar thing for suites.

I think this solves the problem i outlined before, but I'll have to think about it a little more. If I'm right in my thinking here we could regularly run a 'job' on the database to update the gem5_version fields based on non-workload/non-suite resource's manually specified resource_compatibility and schema fields. The job would then automatically update the workload entry's resource_compatibility fields based on it's constituent members' gem5_version fields and, from this, update the gem5_version field for each workload. Then it'd do the same for suites.