Thanks for the report @mouse07410 - this is expected, though as per #617 and #621. fyi, my preference for resolving that is after standardized PQC has landed in openssl master and oqsprovider has a firm target to adapt to.

Thank you - somehow I missed that the OpenSSL default implementation is already merged.

But this brings two questions:

A. Is there any relation between this provider implementation and the code in OpenSSL that implements PQ algorithms?
B. If OpenSSL starts including PQ algorithms by default - what would be the purpose and the need for this "separate" provider? Why would one bother installing it? Is it's only benefit going to be the ability to serve algorithms aliases ("mlkem768" vs. "ML-KEM-768")?

Thank you - somehow I missed that the OpenSSL default implementation is already merged.

The code is not yet merged to main, but the names and OIDs (the former different from what oqsprovider had been using) did get merged already, causing these problems.

A. Is there any relation between this provider implementation and the code in OpenSSL that implements PQ algorithms?

None whatsoever (background at openssl/project#431).

If OpenSSL starts including PQ algorithms by default - what would be the purpose and the need for this "separate" provider?

OpenSSL (now) makes available standard PQ algorithms, while oqsprovider makes available algorithms "in standardization" (new algorithm proposals, different hybridization schemes) -- all with the goal to allow "research and experimentation" (aka, allowing to check their value "in the real world"). And until the time someone makes available an externally loadable provider with the openssl default PQ code for use in OpenSSL versions < 3.5, one may still be tempted to use oqsprovider to "bridge the gap" in older OpenSSL releases. My personal recommendation of course is not to do that for any "serious" use cases as per https://github.com/open-quantum-safe/oqs-provider?tab=readme-ov-file#component-disclaimer. Finally, some people could use it to experiment with algorithms with different non-functional properties (optimizations, different validation properties, etc.)

Why would one bother installing it? Is it's only benefit going to be the ability to serve algorithms aliases ("mlkem768" vs. "ML-KEM-768")?

If one is only interested in standardized PQC that's a very good question and the reason for #610.

Thank you very much for a great clear explanation!

Re. #610 - I concur that there's no real benefit in oqsprovider supporting algorithms that are included in the "default" OpenSSL.

Yes, it looks like for OpenSSL versions < 3.5, oqsprovider will remain a-must.

My remaining question - is there a way to get PQ algorithms working with the current OpenSSL-3.5.0-dev? Or my only (practical) option is to wait until the rest of the PQ code is merged there into the master branch?

My remaining question - is there a way to get PQ algorithms working with the current OpenSSL-3.5.0-dev? Or my only (practical) option is to wait until the rest of the PQ code is merged there into the master branch?

Anything else would mean quite a bit probably "lost work" for me (assuming no one else helps). What's your rationale for using openssl master? Any reason you cannot make do with 3.4.0 for the next couple of days?

What's your rationale for using openssl master?

Making sure my stuff is likely to keep working when 3.5.x gets released, occasionally experiments.

Any reason you cannot make do with 3.4.0 for the next couple of days?

Next couple of days? None whatsoever. Next week or two - tolerable as well, no problem . Somewhat longer - more concerns.

Keeping fingers crossed that everything settles in that time frame -- or I'm finding motivation to give this a try before. I'll hopefully be able to contain your concerns, but just fyi, I'm not employed or remunerated in any shape or form for any of this (and our weather is splendid in the coming days, so I'll probably prefer traveling the Great Outdoors right now), i.e., please allow some leeway -- or send a PR yourself if you find the time. The core problem is probably just a few lines of code that I did wrong a few years ago :-}

As heads-up to @mouse07410 : This looks like it'll take more than a few days. The easiest approach to resolve this (and the one I prefer for several different reasons by now) seems to be to remove support for ML-DSA and ML-KEM from the default build: Would that help you as a "quick fix"?

Would that help you as a "quick fix"?

Alas, no. Because ML-KEM-1024 and ML-DSA-87 are the algorithms that I need.

The current situation is bad, because there is no working ML-DSA and ML-KEM for OpenSSL-3.5.0-dev. OpenSSL (very unwisely, IMHO) decided to "occupy" PQ OIDs, thus preventing other providers like oqsprovider from registering - without actually implementing them.

A reasonable approach for OpenSSL would be to merge either both (PQ provider support and registering OIDs) or neither. Alas, OpenSSL maintainers did not follow it.

@levitte what's your take here?

It is not unwise to register standard OIDs in the base code - this is in fact our normal practise.

We need to handle them in many contexts.
External providers should not operate on the assumption that the base library is unaware of OIDs especially for major algorithms. We broke that assumption knowingly - and did communicate that. The oqs_provider needs to update to handle it - either using the defined NIDa or removing clashing implementations.

The ML-KEM and ML-DSA code are in feature branches
for testing which will be merged mid Feb timeframe.

I don't think oqs_provider changes before that
merge happens necessarily make sense but that is a
decision for the oqs_provider team to make.

Developers are encouraged to check the feature branches out as provide feedback.
ML-KEM is slightly ahead of ML-DSA in terms of completeness but the
code is being actively worked on.

You can use the ML-KEM feature branch and have interoperable
TLS hybrid KEM support in place - that has been operational for
a few weeks.

It is not unwise to register standard OIDs in the base code - this is in fact our normal practise.

Registering is not unwise - but its timing is.

Nobody sane (that I know of) would register something he's not ready to support yet - especially when that something is currently supported by somebody else.

External providers should not operate on the assumption that the base library is unaware of OIDs especially for major algorithms.

What do you expect an external provider to do if it notices that an algorithm is already registered?

Regardless of the answer to the above - the main problem here and now is that those "major algorithms" are registered but not provided.

We broke that assumption knowingly . . .

And I say that it was a very poor decision to register an algorithm you don't support (yet). Common sense suggests that you register when you're ready to support, not before.

. . . - and did communicate that.

Communicating a "non-smart" decision does not make it any better.

You can use the ML-KEM feature branch and have interoperable
TLS hybrid KEM support in place - that has been operational for
a few weeks.

My use case requires both ML-KEM and ML-DSA.

Developers are encouraged to check the feature branches out as provide feedback.

Here's my feedback. Back out the registration part (commit d31fce1972f), split it into pieces according to algorithms they support, and incorporate into ML-KEM, ML-DSA, etc. branches correspondingly. So that when, e.g., you eventually merge ML-KEM branch, it will include both registration and the actual algorithm support.

Summary: the only reasonable approach in this case at this time is to move the registration to the corresponding feature branches that provide the algorithms to be registered. oqsprovider then can start ignoring already-registered OIDs, but the overall codebase won't be broken ("knowingly").

This is a problem because the OBJ database is limited. This issue could have happened if another external provider had started registering its own (conflicting) names for the same OIDs.

Since the very start of designing providers, I saw this problem coming, and argued that we shouldn't have providers depend on the central OBJ database, and that its use should be avoided as much as possible. Unfortunately, OpenSSL doesn't have quite all the replacement parts.

However! The most dependable identity being the OID rather than the name, and that the conflict is on the names, it would seem that if the oqs provider would use the OIDs (in canonical numeric string form, which is available) rather the names to look up entries in the OBJ database, I think it would be rather simple to make things work with minimal effort.

(there's a reason why the OSSL_ALGORITHM name field is supposed to contain as many algorithm identifiers as possible, independent of what names are registered elsewhere)

This is a problem because the OBJ database is limited. This issue could have happened if another external provider had started registering its own (conflicting) names for the same OIDs.

Understood.

Though my venting is because the OpenSSL maintainers' decision "knowingly" broke running code, which was unnecessary.

Since the very start of designing providers, I saw this problem coming, and argued that we shouldn't have providers depend on the central OBJ database, and that its use should be avoided as much as possible. Unfortunately, OpenSSL doesn't have quite all the replacement parts.

And right you were. :-(

The most dependable identity being the OID rather than the name, and that the conflict is on the names, it would seem that if the oqs provider would use the OIDs (in canonical numeric string form, which is available) rather the names to look up entries in the OBJ database, I think it would be rather simple to make things work with minimal effort.

Thank you for the suggestion! @baentsch are you willing to try? Because it seems that eventually this would have to be done, no matter what.

Already ongoing, courtesy @levitte in #629...

To summarize the problem here:

• OpenSSL "claims" to provide support for NIST PQ standards (ML-KEM & ML-DSA), which prevents less-nimble external providers from registering their support for them; and at the same time
• OpenSSL does not actually support those algorithms.

Remedy

OpenSSL needs to delay "claiming support" until it is ready to merge the actual implementation.

Reason: even if the provider "recognized" that somebody else laid a claim on those, and did not register, e.g., ML-KEM - a user who needs ML-KEM, will not get it.

Comments?

I don't quite agree: OpenSSL registered standarized IDs and oqsprovider fails to properly handle this. This is correctly tagged "bug" of oqsprovider and we have an (this) issue tracking this.

I see this as a temporary problem and one that will be fixed in due course: For now, oqsprovider keeps working with all released openssl versions and it is my goal to achieve that again with the next release. I just do not want to waste cycles building/testing a possible fix against 4 different versions of openssl (master and the 3 PQC feature branches) OR fixing the problem possibly 4 times (to master now and again the moment each feature branch merges):

Thus, IMO this is more a problem of my "laziness" (not being willing to work for free more often than seems necessary) and/or proof of the lack of a community supporting oqsprovider/ willing to spend time providing a complete fix before I do. And honestly, I can understand that lack of support: Why work on getting something to work that's becoming irrelevant (providing support for standardized PQC algs) anyway? I will do it as I created oqsprovider and feel I owe it to its users -- but at my pace. All I could do is apologize to those with a different pace or expectations.

I don't quite agree: OpenSSL registered standarized IDs and oqsprovider fails to properly handle this. This is correctly tagged "bug" of oqsprovider and we have an (this) issue tracking this.

OK, no argument here. But please, would you mind explaining how oqsprovider should handle this situation? When OpenSSL advertises ML-KEM, but doesn't implement it?

And how should a user invoke ML-KEM in this case (assuming oqsprovider is fully-fixed and works perfectly)?

I see this as a temporary problem and one that will be fixed in due course

Yes, I'm sure of that. E.g., when ml-kem and ml-dsa branches are merged, the problem will cease to exist, at least for me.

All I could do is apologize to those with a different pace or expectations.

No problem, and no apologies are necessary. But I'd like to see the answer to my question at the head of this post: how should an external provider behave when some of his algorithms are claimed by somebody else?

But please, would you mind explaining how oqsprovider should handle this situation? When OpenSSL advertises ML-KEM, but doesn't implement it?

And how should a user invoke ML-KEM in this case (assuming oqsprovider is fully-fixed and works perfectly)?

I'd say, as before (using it's "convenient name", e.g., mlkem768 -- but additionally then, the "standardized name" ML-KEM-768) -- please see/comment on the design proposal.

how should an external provider behave when some of his algorithms are claimed by somebody else?

That seems like a very sensible question to be asked and answered in the openssl discussion board for "posterity" (and/or maybe trigger a documentation update for provider authors).

My opinion is that in general a provider should provide at the very least a warning message, but not cease to operate (with/for other algorithms).

Anything else is up to the provider's intentions: For oqsprovider, dependent on the outcome of this discussion, it may outright cease to provide these algorithms and/or still do provide them (e.g., based on a config flag, such as for the unwary user to have to know exactly what s/he's doing: "I really don't want to use the reliable default/fips provider code but experimental/research PQC").

But thinking this through, this triggers an entirely new question in my mind: Might the provider be enticed to use another provider's functionality, e.g., to provide additional features (other experimental hybrid and composites come to mind in this context)? Created #631 to track/discuss.

To summarize the problem here:

• OpenSSL "claims" to provide support for NIST PQ standards (ML-KEM & ML-DSA), which prevents less-nimble external providers from registering their support for them; and at the same time

No - OpenSSL does not claim to support implementations of the PQ algorithms.
The algorithm support mechanism is entirely separate from the registration of object identifiers.
OpenSSL has a large list of identifiers where a object identifier to name mapping is provided. These have nothing to do with whether or not the functionality represented by the underlying algorithm is present.

OpenSSL is simply aware of the object identifiers - that is all.

In the feature branches we have implementations which will be merged pre-alpha release (assuming they pass the review criteria for inclusion).

The oqs_provider can be changed to allow for the object identifier table being aware of the object mappings or it can remove its clashing entries. Either approach works. Or the oqs_provider can work off any of the earlier releases of OpenSSL.

Once the feature branches make it back into the main tree, there will be multiple implementations (if you have the default or fips providers and the oqs_provider) and you can select which provider is used in property query strings (by indicating a preference for a particular provider). The concept of multiple implementations existing for an algorithm is part of the core concept supported by the provider interface.

Thanks,