[xmlsec] Signing on Windows using SHA-256 hash

[Miklos Vajna]

Hi,

The context is that LibreOffice currently bundles (a patched) xmlsec
1.2.14, and I'm trying to bring our xmlsec version up to date step by
step. This would result in dropping no longer necessary backports and
would give me a clear picture on what patches are still relevant for
upstream xmlsec.

As part of this effort, I rebased all our patches against xmlsec 1.2.15,
which is already a nontrivial change: as far as I see this was the first
version adding support for the SHA2 family (SHA-256, etc.)

Our current xmlsec 1.2.14 has patches for the nss and mscrypto backend
to support SHA-256, my hope is that we can drop these, as 1.2.15
supports these out of the box.

When I added the 1.2.14 SHA-256 patches, the result was:

https://cgit.freedesktop.org/libreoffice/core/tree/external/libxmlsec/xmlsec1-nss-sha256.patch.1

and

https://cgit.freedesktop.org/libreoffice/core/tree/external/libxmlsec/xmlsec1-mscrypto-sha256.patch.1

(I don't think this is interesting, but in case it is: the current
1.2.15 patches are here:

https://cgit.freedesktop.org/~vmiklos/lo-core/tree/external/libxmlsec?h=xmlsec

They are not in upstream LibreOffice yet, as the upgrade currently would
break Windows OOXML signing, see below.)

The NSS part is fairly trivial and I tested that 1.2.15 + our patches
(without any custom SHA-256 patch) works fine. This is great for Linux /
OS X.

However, the Windows / mscrypto backend is more problematic. Again some
background: when I did the 1.2.24 SHA-256 patch for mscrypto, I had to
solve two nontrivial problems:

1) The PROV_RSA_FULL crypto provider doesn't support SHA-256, one needs
PROV_RSA_AES. I assume this is also taken care of in 1.2.15, as a nice
list of crypto providers is there to gracefully fall back to
PROV_RSA_FUll, etc. This makes verification of RSA+SHA-256 signatures
working.

2) A more complex problem is the signing itself, still using
RSA+SHA-256. The root of the problem is that in case the private key is
loaded from a PROV_RSA_FULL provider, then we would use the hProv of
that provider, and that won't work with SHA-256. The above linked
mscrypto patch solves this by opening a separate crypto context just for
signing, see code from

https://cgit.freedesktop.org/libreoffice/core/tree/external/libxmlsec/xmlsec1-mscrypto-sha256.patch.1#n246

Now the problem. :-)

If i just drop our two sha256 patches and I try signing using RSA+SHA256
on Windows, then I get this:

errorcallback.cxx:43: ..\src\mscrypto\signatures.c:485: xmlSecMSCryptoSignatureExecute() '' 'CryptCreateHash' 4 ' '
errorcallback.cxx:43: ..\src\transforms.c:2208: xmlSecTransformDefaultPushBin() 'rsa-sha256' 'xmlSecTransformExecute' 1 'final=0'
errorcallback.cxx:43: ..\src\transforms.c:2870: xmlSecTransformIOBufferWrite() 'rsa-sha256' 'xmlSecTransformPushBin' 1 ' '
I/O error : flush error
C14N error : Internal error : flushing output buffer
errorcallback.cxx:43: ..\src\c14n.c:457: xmlSecTransformC14NExecute() 'c14n' 'xmlC14NExecute' 5 ' '
errorcallback.cxx:43: ..\src\c14n.c:268: xmlSecTransformC14NPushXml() 'c14n' 'xmlSecTransformC14NExecute' 1 ' '
errorcallback.cxx:43: ..\src\mscrypto\signatures.c:485: xmlSecMSCryptoSignatureExecute() '' 'CryptCreateHash' 4 ' '
errorcallback.cxx:43: ..\src\transforms.c:2208: xmlSecTransformDefaultPushBin() 'rsa-sha256' 'xmlSecTransformExecute' 1 'final=1'
errorcallback.cxx:43: ..\src\transforms.c:2892: xmlSecTransformIOBufferClose() 'rsa-sha256' 'xmlSecTransformPushBin' 1 ' '
errorcallback.cxx:43: ..\src\transforms.c:1237: xmlSecTransformCtxXmlExecute() '' 'xmlSecTransformPushXml' 1 'transform=c14n'
errorcallback.cxx:43: ..\src\xmldsig.c:614: xmlSecDSigCtxProcessSignatureNode() '' 'xmlSecTransformCtxXmlExecute' 1 ' '
errorcallback.cxx:43: ..\src\xmldsig.c:303: xmlSecDSigCtxSign() '' 'xmlSecDSigCtxSigantureProcessNode' 1 ' '

GetLastError() returns 80090008 (NTE_BAD_ALGID).

And indeed if I query the name of the provider, then I get: 'Microsoft
Enhanced Cryptographic Provider v1.0', i.e. the FULL one, not the AES
one, so I understand why SHA-256 won't fly.

I tried to do the same trick in 1.2.15 as in the above patch, to create
a new crypto context using PROV_RSA_AES and the PP_CONTAINER of the old
context, but on 1.2.15 this doesn't seem to help: I still get the
non-AES name when I query the provider name of the new context. So I'm
currently out of ideas how to proceed. :-)

I checked the git 1.2.15..master git history of the mscrypto backend,
and the only relevant fix I found was
c6e41273caf35d3d0e4084204c83019c1d25d832 (remove SILENT flag, it breaks
some CSPs including default MS CSPs, 2011-09-06), but backporting that
fix did not change anything.

One more detail, I'm testing this on a Windows 7, and as I said 1.2.14 +
my poor SHA-256 patches did work on this system, so I assume it's
something on the xmlsec side.

Any ideas? How is RSA+SHA-256 signing (not verification) supposed to
work on Windows?

Thanks a lot,

Miklos
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://www.aleksey.com/pipermail/xmlsec/attachments/20160308/e15486b0/attachment.sig>