[xmlsec] signing thousands of files - too many open files

[Dimitrios Siganos]

Hi,

I have a need to sign and verify thousands of files using xmldsig.
Unfortunately, I can't archive them and sign the archive.

This is my setup:
* 2000 files (file1, file2, ..., file2000)
* a signature template that references all of the 2000 files (i.e. 2000
file references)
<Signature>
  <SignedInfo>
    <CanonicalizationMethod Algorithm="http://www.w3.org/2006/12/xml-c14n11
"/>
    <SignatureMethod Algorithm="
http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/>
    <Reference URI="file1">
      <DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
      <DigestValue/>
    </Reference>
    ... 1998 more references ...
    <Reference URI="file2000">
      <DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
      <DigestValue/>
    </Reference>
  </SignedInfo>
  <SignatureValue/>
  <KeyInfo><X509Data><X509Certificate/></X509Data></KeyInfo>
</Signature>

Running xmlsec1 --sign fails with the error "Too many open files". The
reason is that libxmlsec opens all 2000 files, and leaves them open, until
the end of the life of the entire operation.

I don't have the option to adjust the open files limit (ulimit).

I am able to fix/workaround the "too many open files" problem by closing
the input files as soon as we are finished reading from them. This is the
gist of my change:

diff --git a/src/transforms.c b/src/transforms.c
index 8a2ded2..fa5b885 100644
--- a/src/transforms.c
+++ b/src/transforms.c
@@ -1195,6 +1195,7 @@ xmlSecTransformCtxUriExecute(xmlSecTransformCtxPtr
ctx, const xmlChar* uri) {
         return(-1);
     }

+    xmlSecTransformInputURIFinalize(uriTransform);
     ctx->status = xmlSecTransformStatusFinished;
     return(0);
 }

Could you please comment on whether my change makes sense and is correct?

Regards,
Dimitrios Siganos
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.aleksey.com/pipermail/xmlsec/attachments/20150302/813744d7/attachment.html>