[xmlsec] Proposed patch to allow OpenSSL/ENGINE operations

[Erwann ABALEA]

Bonjour,

While working on integrating PKCS#11 support for OpenSSL for later use
by xmlsec, I found that the xmlSecOpenSSLKeyDataRsaGetType function
reports the type of the key using a rather naive method. This function
tries to make the difference between a public and private key by
checking wether the private exponent exists. Unfortunately, this is
not a good way to do it for ENGINE's managed private keys, where the
private key is held in a hardware token. The result is that such a key
is declared as a public one, and thus can't be used for signing or
decrypting operations.

I'm proposing a small patch to allow such keys to be used. In fact,
the patch returns that a key can be used for public and private
operations. That may be bad, but:
 - I have found no way to check that a given RSA* key can be used for
   private operations
 - if you try to sign/decrypt something with a pure public key
   declared as private by this patch, you'll still have an error, but
   later, so it is catched by the xmlsec library.

Here it is:
-----
diff -Naur xmlsec1-1.2.8/src/openssl/evp.c xmlsec1-1.2.8.new/src/openssl/evp.c
--- xmlsec1-1.2.8/src/openssl/evp.c	2004-03-17 06:06:46.000000000 +0100
+++ xmlsec1-1.2.8.new/src/openssl/evp.c	2005-07-07 16:49:05.000000000 +0200
@@ -1496,11 +1496,22 @@
     
     rsa = xmlSecOpenSSLKeyDataRsaGetRsa(data);
     if((rsa != NULL) && (rsa->n != NULL) && (rsa->e != NULL)) {
+        /* TODO: Find a better way to check if a given key can be used
+	 * for private operations (sign or decrypt). Checking for the
+	 * presence of the private exponent is not a good solution, as
+	 * you can not have it for ENGINE's managed keys.
+	 * Right now, tell libxmlsec that the key is usable for both
+	 * purposes, and let OpenSSL properly return an error if the
+	 * given key can not be used as wanted.
+	 */
+        return(xmlSecKeyDataTypePrivate | xmlSecKeyDataTypePublic);
+/*
 	if(rsa->d != NULL) {
 	    return(xmlSecKeyDataTypePrivate | xmlSecKeyDataTypePublic);
 	} else {
 	    return(xmlSecKeyDataTypePublic);
 	}
+*/
     }
 
     return(xmlSecKeyDataTypeUnknown);
-----

Right now, I can't find any easy way to add transparent support for
OpenSSL's ENGINEs, but from a xmlsec-user point of view, it's
relatively easy to do. I can provide examples if necessary (not now,
I'll go on vacation very soon).


-- 
Erwann ABALEA <erwann.abalea at keynectis.com>