X509::Certificate#== considers certs with different serial numbers and common names to be equal

[gongfarmer]

Minimal reproduction:

#!/usr/bin/ruby

require 'openssl'

include OpenSSL::X509

cert1, cert2 = 2.times.map { Certificate.new }

cert1.serial=1
cert1.subject=Name.new([["CN","ONE"]])
cert1.issuer =  Name.new([["CN","ONE_ISSUER"]])


cert2.serial=2
cert2.subject=Name.new([["CN","TWO"]])
cert2.issuer = Name.new([["CN","TWO_ISSUER"]])


### Given that the serial and cn are different, these should not be equal, but they are anyway
if cert1 == cert2
  puts "equal"
else
  puts "not equal"
end

According to the docs:

= OpenSSL::X509::Certificate#==

(from ruby core)
=== Implementation from Certificate
------------------------------------------------------------------------
  cert1 == cert2 -> true | false

------------------------------------------------------------------------

Compares the two certificates. Note that this takes into account all
fields, not just the issuer name and the serial number.

Tested on these versions:

openssl gem 3.2.0 on debian 12 with ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux-gnu]
openssl gem 3.3.0 on OS X 15.2 with ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +PRISM [arm64-darwin24]

[rhenium]

OpenSSL::X509::Certificate#== wraps X509_cmp(): https://github.com/openssl/openssl/blob/cfc62a3c467b73207315060de1fe9af90c82d863/crypto/x509/x509_cmp.c#L155-L182

It appears that this results in a zero (equal) return when the certificates are invalid/incomplete and it can't get a DER encoding value.