[xmlsec] Problem with some cert which has a negative serial number

[Chandler Peng]

Hello , Aleksey ,
 
We found there will be failed using some certificate to sign when these 
certificate have one negative integer in  the serial number field .  we 
will get an KEY_NOT_FOUND error indicate the cert or the privatekey can 
not found in the cert store or the key store . we think there is the 
cert not found when searching the cert with issuer name and serial 
number . At the begining , we got the raw strings of IssuerName(DN) and 
SerialNumber(SN) from the certificate without any change . Then we 
passed the DN and SN to the libxmlsec . The SN has been changed from der 
format to decimal format in xmlSecBnToString() and there are no sign to 
record whether the integer is negative or not.  So the correct der 
string  can not come back from decimal string when the integer is 
negative and this cause searching cert process failed when using  the DN 
and the 'wrong' SN as the parameter of CertFindCertificateInStore().

According  to RFC3280 , the serial number MUST be a positive integer 
assigned by the CA to each certificate, BUT according to X509 , a serial 
number in certificate can be positive or negative.

"RFC 3280 mandates that serial numbers be positive integers that are at 
most 20 octets long, but X.509 simply states that serial numbers are 
integers.  So, if a certificate with a negative serial number is not 
incorrect, it simply was not generated in a PKIX compliant manner."
--from NIST, the author org of X509.

So , what should we do on this scene ? Does the libxmlsec will support 
these certificate in the coming version?

--Chandler Peng.