[xmlsec] signing thousands of files - too many open files

[Aleksey Sanin]

Dimitrios,

Could you please try out the current MASTER in git? I've
put in-place a slightly different fix (same idea though):

To ssh://aleksey@git.gnome.org/git/xmlsec
   f26e572..17a07fd  master -> master

Thanks again for reporting the issue!

Best,

Aleksey

On 3/2/15 8:53 AM, Aleksey Sanin wrote:
> I would start from editing /etc/security/limits.conf or
> /etc/sysctl.conf to increase the limit :)
> 
> I need to think about this change. It looks safe but
> somewhat breaks the abstraction level. I wonder if there
> is a better way to do it.
> 
> Thanks!
> 
> Aleksey
> 
> On 3/2/15 3:36 AM, Dimitrios Siganos wrote:
>> Hi,
>>
>> I have a need to sign and verify thousands of files using xmldsig.
>> Unfortunately, I can't archive them and sign the archive.
>>
>> This is my setup:
>> * 2000 files (file1, file2, ..., file2000)
>> * a signature template that references all of the 2000 files (i.e. 2000
>> file references)
>> <Signature>
>>   <SignedInfo>
>>     <CanonicalizationMethod
>> Algorithm="http://www.w3.org/2006/12/xml-c14n11"/>
>>     <SignatureMethod
>> Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/>
>>     <Reference URI="file1">
>>       <DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
>>       <DigestValue/>
>>     </Reference>
>>     ... 1998 more references ...
>>     <Reference URI="file2000">
>>       <DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
>>       <DigestValue/>
>>     </Reference>
>>   </SignedInfo>
>>   <SignatureValue/>
>>   <KeyInfo><X509Data><X509Certificate/></X509Data></KeyInfo>
>> </Signature>
>>
>> Running xmlsec1 --sign fails with the error "Too many open files". The
>> reason is that libxmlsec opens all 2000 files, and leaves them open,
>> until the end of the life of the entire operation.
>>
>> I don't have the option to adjust the open files limit (ulimit).
>>
>> I am able to fix/workaround the "too many open files" problem by closing
>> the input files as soon as we are finished reading from them. This is
>> the gist of my change:
>>
>> diff --git a/src/transforms.c b/src/transforms.c
>> index 8a2ded2..fa5b885 100644
>> --- a/src/transforms.c
>> +++ b/src/transforms.c
>> @@ -1195,6 +1195,7 @@ xmlSecTransformCtxUriExecute(xmlSecTransformCtxPtr
>> ctx, const xmlChar* uri) {
>>          return(-1);
>>      }
>>  
>> +    xmlSecTransformInputURIFinalize(uriTransform);
>>      ctx->status = xmlSecTransformStatusFinished;
>>      return(0);
>>  }
>>
>> Could you please comment on whether my change makes sense and is correct?
>>
>> Regards,
>> Dimitrios Siganos
>>
>>
>>
>> _______________________________________________
>> xmlsec mailing list
>> xmlsec at aleksey.com
>> http://www.aleksey.com/mailman/listinfo/xmlsec
>>
> _______________________________________________
> xmlsec mailing list
> xmlsec at aleksey.com
> http://www.aleksey.com/mailman/listinfo/xmlsec
>