How to handle JSON schema validation

[kurtseifried]

So we want to validate our JSON data where possible. Many (e.g. OSV) have schemas. However in order to properly use $ref links in schema (essentially you can point at other schemas from within a schema) the schema data needs to be served as "application/schema+json" so for example we can't just point at the GitHub "raw" links because they get served as "text/plain"

So we have two main options as I see it:

1. Find official endpoints that serve the schemas we need as "application/schema+json"
2. Create our own endpoint to serve schema files as "application/schema+json"

Ideally each schema we rely upon (OSV, CVE. NIST, etc.) would have an endpoint that works, most (all?) do not.

This also doesn't address longer ter issues like projects using the OSV data format but in YAML files for example.

I'd like to propose (conceptually) we have something like "schemas.gsd.id" that serves various schemas (JSON, YAML, etc.). Long term it could even provide an API that validates data. We would need to harvest up all the schemas and versions we use and serve them, or maybe simply provide a "passthrough" API that has an allow list of things to serve and takes a request like:

https://schema.gsd.id/raw.githubusercontent.com/ossf/osv-schema/v1.4.0/validation/schema.json

and goes and gets https://raw.githubusercontent.com/ossf/osv-schema/v1.4.0/validation/schema.json and then serves it with "application/schema+json"

we would then need to change lines like:

"$ref": "https://raw.githubusercontent.com/ossf/osv-schema/v1.4.0/validation/schema.json"

to

"$ref": "https://schema.gsd.id/raw.githubusercontent.com/ossf/osv-schema/v1.4.0/validation/schema.json"

and it would work. Thoughts/comments?

[joshbuker]

Another option that would be sane, imo, is putting this under api.gsd.id

e.g.

• https://api.gsd.id/v1/schemas/osv/v1.4.0
• https://api.gsd.id/v1/schemas/cve/v4
• https://api.gsd.id/v1/schemas/cve/v5

etc

[joshbuker]

Another benefit of doing it this way, is we could potentially make some of these very minimal wrappers around the githubcontent source, requiring less long-term maintenance.

[joshbuker]

The ideal solution though, would be getting the upstream projects to provide endpoints in the appropriate format. Which also would be ideal for the $id field of the respective schemas. I saw you already made ossf/osv-schema#126 for OSV, but we should also do the same for CVE, NVD, CISA, etc where possible.

[joshbuker]

Alternatively, can also see if the validators can be updated to support text/plain $refs, perhaps by passing a flag to the validator class.

[kurtseifried]

So here's the thing:

1. Updating all the schemas to provide proper endpoints will never be 100%, so we'll have to have some solution for those that don't
2. Updating all the schema parsing software to support text/plain, or even a handful of parsers that we use will never be 100%, so we'll have to have some solution for those that don't

which logically leads to:

3. We do control the data we provide and this is the easiest place to make a change. We can get 100% coverage here.

which then leads to "how exactly"?

Hosting it ourselves with correct content type means we don't have to fix any software. That's a huge bonus.

Hosting it with the full path, prepended with scha.gsd.id or api.gsd.id or whatever means we don't lose any data, and we can change it easily without having to worry (e.g. multiple things can map to https://api.gsd.id/v1/schemas/cve/v5 but only https://schema.gsd.id/raw.githubusercontent.com/ossf/osv-schema/v1.4.0/validation/schema.json maps to https://raw.githubusercontent.com/ossf/osv-schema/v1.4.0/validation/schema.json

[joshbuker]

Having schema.gsd.id be a super simple Cloudflare worker that just transparently re-serves as json schema content type seems like the easiest and most intuitive solution. Created #177