[xmlsec] Signing on Windows using SHA-256 hash

[Aleksey Sanin]

Milkos,

First, as I mentioned before, I would be happy to merge all the
upstream patches that makes sense for the main xmlsec. Please
don't hesitate to send pull requests :)

Unfortunately, I don't have a Windows environment anymore so I can't
debug it myself. But I can give you a suggestion. As far as I recall,
there is a function xmlSecMSCryptoFindProvider() which is used to
find out the crypto provider for specific operation. I would suggest
to put a breakpoint on this function and compare parameters for
the cases of signing and verification.

Let me know if it makes sense at all!

Aleksey

On 3/8/16 1:47 AM, Miklos Vajna wrote:
> Hi,
> 
> The context is that LibreOffice currently bundles (a patched) xmlsec
> 1.2.14, and I'm trying to bring our xmlsec version up to date step by
> step. This would result in dropping no longer necessary backports and
> would give me a clear picture on what patches are still relevant for
> upstream xmlsec.
> 
> As part of this effort, I rebased all our patches against xmlsec 1.2.15,
> which is already a nontrivial change: as far as I see this was the first
> version adding support for the SHA2 family (SHA-256, etc.)
> 
> Our current xmlsec 1.2.14 has patches for the nss and mscrypto backend
> to support SHA-256, my hope is that we can drop these, as 1.2.15
> supports these out of the box.
> 
> When I added the 1.2.14 SHA-256 patches, the result was:
> 
> https://cgit.freedesktop.org/libreoffice/core/tree/external/libxmlsec/xmlsec1-nss-sha256.patch.1
> 
> and
> 
> https://cgit.freedesktop.org/libreoffice/core/tree/external/libxmlsec/xmlsec1-mscrypto-sha256.patch.1
> 
> (I don't think this is interesting, but in case it is: the current
> 1.2.15 patches are here:
> 
> https://cgit.freedesktop.org/~vmiklos/lo-core/tree/external/libxmlsec?h=xmlsec
> 
> They are not in upstream LibreOffice yet, as the upgrade currently would
> break Windows OOXML signing, see below.)
> 
> The NSS part is fairly trivial and I tested that 1.2.15 + our patches
> (without any custom SHA-256 patch) works fine. This is great for Linux /
> OS X.
> 
> However, the Windows / mscrypto backend is more problematic. Again some
> background: when I did the 1.2.24 SHA-256 patch for mscrypto, I had to
> solve two nontrivial problems:
> 
> 1) The PROV_RSA_FULL crypto provider doesn't support SHA-256, one needs
> PROV_RSA_AES. I assume this is also taken care of in 1.2.15, as a nice
> list of crypto providers is there to gracefully fall back to
> PROV_RSA_FUll, etc. This makes verification of RSA+SHA-256 signatures
> working.
> 
> 2) A more complex problem is the signing itself, still using
> RSA+SHA-256. The root of the problem is that in case the private key is
> loaded from a PROV_RSA_FULL provider, then we would use the hProv of
> that provider, and that won't work with SHA-256. The above linked
> mscrypto patch solves this by opening a separate crypto context just for
> signing, see code from
> 
> https://cgit.freedesktop.org/libreoffice/core/tree/external/libxmlsec/xmlsec1-mscrypto-sha256.patch.1#n246
> 
> Now the problem. :-)
> 
> If i just drop our two sha256 patches and I try signing using RSA+SHA256
> on Windows, then I get this:
> 
> errorcallback.cxx:43: ..\src\mscrypto\signatures.c:485: xmlSecMSCryptoSignatureExecute() '' 'CryptCreateHash' 4 ' '
> errorcallback.cxx:43: ..\src\transforms.c:2208: xmlSecTransformDefaultPushBin() 'rsa-sha256' 'xmlSecTransformExecute' 1 'final=0'
> errorcallback.cxx:43: ..\src\transforms.c:2870: xmlSecTransformIOBufferWrite() 'rsa-sha256' 'xmlSecTransformPushBin' 1 ' '
> I/O error : flush error
> C14N error : Internal error : flushing output buffer
> errorcallback.cxx:43: ..\src\c14n.c:457: xmlSecTransformC14NExecute() 'c14n' 'xmlC14NExecute' 5 ' '
> errorcallback.cxx:43: ..\src\c14n.c:268: xmlSecTransformC14NPushXml() 'c14n' 'xmlSecTransformC14NExecute' 1 ' '
> errorcallback.cxx:43: ..\src\mscrypto\signatures.c:485: xmlSecMSCryptoSignatureExecute() '' 'CryptCreateHash' 4 ' '
> errorcallback.cxx:43: ..\src\transforms.c:2208: xmlSecTransformDefaultPushBin() 'rsa-sha256' 'xmlSecTransformExecute' 1 'final=1'
> errorcallback.cxx:43: ..\src\transforms.c:2892: xmlSecTransformIOBufferClose() 'rsa-sha256' 'xmlSecTransformPushBin' 1 ' '
> errorcallback.cxx:43: ..\src\transforms.c:1237: xmlSecTransformCtxXmlExecute() '' 'xmlSecTransformPushXml' 1 'transform=c14n'
> errorcallback.cxx:43: ..\src\xmldsig.c:614: xmlSecDSigCtxProcessSignatureNode() '' 'xmlSecTransformCtxXmlExecute' 1 ' '
> errorcallback.cxx:43: ..\src\xmldsig.c:303: xmlSecDSigCtxSign() '' 'xmlSecDSigCtxSigantureProcessNode' 1 ' '
> 
> GetLastError() returns 80090008 (NTE_BAD_ALGID).
> 
> And indeed if I query the name of the provider, then I get: 'Microsoft
> Enhanced Cryptographic Provider v1.0', i.e. the FULL one, not the AES
> one, so I understand why SHA-256 won't fly.
> 
> I tried to do the same trick in 1.2.15 as in the above patch, to create
> a new crypto context using PROV_RSA_AES and the PP_CONTAINER of the old
> context, but on 1.2.15 this doesn't seem to help: I still get the
> non-AES name when I query the provider name of the new context. So I'm
> currently out of ideas how to proceed. :-)
> 
> I checked the git 1.2.15..master git history of the mscrypto backend,
> and the only relevant fix I found was
> c6e41273caf35d3d0e4084204c83019c1d25d832 (remove SILENT flag, it breaks
> some CSPs including default MS CSPs, 2011-09-06), but backporting that
> fix did not change anything.
> 
> One more detail, I'm testing this on a Windows 7, and as I said 1.2.14 +
> my poor SHA-256 patches did work on this system, so I assume it's
> something on the xmlsec side.
> 
> Any ideas? How is RSA+SHA-256 signing (not verification) supposed to
> work on Windows?
> 
> Thanks a lot,
> 
> Miklos
> 
> 
> 
> _______________________________________________
> xmlsec mailing list
> xmlsec at aleksey.com
> http://www.aleksey.com/mailman/listinfo/xmlsec
>