[xmlsec] Proposed patch to allow OpenSSL/ENGINE operations

[Erwann ABALEA]

Hodie Non. Iul. MMV est, Aleksey Sanin scripsit:
> >I haven't asked, but checked the OpenSSL 0.9.8 release, and no,
> >there's still no way to check if a key can be used for a private
> >operation, other than doing it.
> It might be a good idea to ask. Even if it is not there now this
> might give openssl developers a reason to add it in the future.

I'll ask. Not now, because I won't be here for the next 2 weeks, and
don't want to start a request without being able to answer
complementary questions.

> >Anyway. Imagine you're using the CAPI stack, with its native support
> >for hardware tokens. You get a handle on a private key, declared as
> >such. But when you want to perform the private operation, the token is
> >removed. The CAPI then returns an error, and it is properly catched by
> >xmlsec, right?
> >What is the difference with the proposed behaviour introduced by my
> >patch?
> >
> True. But if application does operate only with keys in memory and
> does not want to know about hardware tokens, then it expects to get back
> real private key when it asks for one.

OK. What about this patch, then?

-----
diff -Naur xmlsec1-1.2.8/src/openssl/evp.c xmlsec1-1.2.8.new/src/openssl/evp.c
--- xmlsec1-1.2.8/src/openssl/evp.c	2004-03-17 06:06:46.000000000 +0100
+++ xmlsec1-1.2.8.new/src/openssl/evp.c	2005-07-08 10:39:33.000000000 +0200
@@ -1496,10 +1496,26 @@
     
     rsa = xmlSecOpenSSLKeyDataRsaGetRsa(data);
     if((rsa != NULL) && (rsa->n != NULL) && (rsa->e != NULL)) {
-	if(rsa->d != NULL) {
-	    return(xmlSecKeyDataTypePrivate | xmlSecKeyDataTypePublic);
-	} else {
-	    return(xmlSecKeyDataTypePublic);
+        if(rsa->engine != NULL)
+            /* OpenSSL provides no way to determine that a given RSA key 
+	     * is able to perform private or public operations other than 
+	     * trying to do this operation.
+	     * In case an ENGINE is associated with such a key, we return 
+	     * to the caller that the key is usable for both purposes,
+	     * the real test will be done by the OpenSSL+ENGINE stack,
+	     * and any error will be returned and catched by xmlSec
+	     * properly.
+	     * If no ENGINE is associated with it (99% of the time,
+	     * with pure software keys), we rely on the original
+	     * check, naive but functional.
+	     */
+            return(xmlSecKeyDataTypePrivate | xmlSecKeyDataTypePublic);
+	else {
+	    if(rsa->d != NULL) {
+	        return(xmlSecKeyDataTypePrivate | xmlSecKeyDataTypePublic);
+    	    } else {
+	        return(xmlSecKeyDataTypePublic);
+	    }
 	}
     }
-----

Feel free to modify/adjust the coding style, as I don't use yours :)

-- 
Erwann ABALEA <erwann.abalea at keynectis.com>