[xmlsec] Support for really large XML documents

[Aleksey Sanin]

Unfortunately, I have to have the whole document in memory for C14N

Aleksey

On 5/24/12 11:08 AM, Vit Zikmund wrote:
> Hi Aleksey, thanks for the tip.
> I've tried it, but apparently, it's not the case. I've debugged the code
> and found the source of the error.
> Here
> _http://git.gnome.org/browse/xmlsec/tree/src/c14n.c#n277_xmlOutputBufferClose(buf)returns
> negative number, but it's not an error code - it's an overflowed byte
> counter.
> The overflow happens without error during the transformation execution
> in the libxml2 code - at the end of *xmlOutputBufferWrite*() (
> _http://git.gnome.org/browse/libxml2/tree/xmlIO.c#n3445_).
> Everything is just an 'int' over there. If I add a line checking for
> overflow to keep the value positive, my test passes, but that is some
> nasty hack.
> 
> I've already contacted the author and he said such big value shouldn't
> ever be there and suggested this might be a bad design.
> This is the thread on libxml mailing list:
> _https://mail.gnome.org/archives/xml/2012-May/msg00075.html_
> 
> Can you comment on that? Might this be related to your comment few lines
> above the error saying:
> /* we are using a _semi_-hack here: we know that xmlSecPtrList keeps
>  * all pointers in the big array */
> 
> Thanks again,
> Vit
> 
> Might this be somehow related to the comment few lines above
> 
> Aleksey Sanin <aleksey at aleksey.com> wrote on 05/23/2012 09:28:11 PM:
> 
>> The error indicates that we can't allocate output buffer correctly. If
>> I would guess, then I would see if the "size" parameter is treated as
>> negative number when it exceeds 2G.
>>
>> Try to change include/xmlsec/xmlsec.h and change the xmlSecSize to be
>> a typedef to size_t all the time (dont' forget to recompile xmlsec
>> after this change).
>>
>> Aleksey
>>
>> On 5/23/12 8:28 AM, Vit Zikmund wrote:
>> > Hello,
>> > we are trying to use the XMLSec utility to verify documents signed with
>> > our own application and probably have hit a limit of the document size,
>> > that XMLSec is able to process.
>> >
>> > The simplest question is: Does XMLSec support handling large
>> > documents/files? Is is about to? For large I mean 2 gigabytes and more.
>> >
>> > I can verify a document of 1GB, but little over 2GB returns an error:
>> >
>> >
>>
> func=xmlSecTransformC14NPushXml:file=c14n.c:line=279:obj=c14n:subj=xmlOutputBufferClose:error=5:libxml2
>> > library function failed:
>> >
>>
> func=xmlSecTransformDefaultPushXml:file=transforms.c:line=2417:obj=enveloped-
>> signature:subj=xmlSecTransformPushXml:error=1:xmlsec
>> > library function failed:
>> >
>>
> func=xmlSecTransformCtxXmlExecute:file=transforms.c:line=1236:obj=unknown:subj=xmlSecTransformPushXml:error=1:xmlsec
>> > library function failed:transform=enveloped-signature
>> >
>>
> func=xmlSecTransformCtxExecute:file=transforms.c:line=1296:obj=unknown:subj=xmlSecTransformCtxXmlExecute:error=1:xmlsec
>> > library function failed:
>> >
>>
> func=xmlSecDSigReferenceCtxProcessNode:file=xmldsig.c:line=1571:obj=unknown:subj=xmlSecTransformCtxExecute:error=1:xmlsec
>> > library function failed:
>> >
>>
> func=xmlSecDSigCtxProcessSignedInfoNode:file=xmldsig.c:line=804:obj=unknown:subj=xmlSecDSigReferenceCtxProcessNode:error=1:xmlsec
>> > library function failed:node=Reference
>> >
>>
> func=xmlSecDSigCtxProcessSignatureNode:file=xmldsig.c:line=547:obj=unknown:subj=xmlSecDSigCtxProcessSignedInfoNode:error=1:xmlsec
>> > library function failed:
>> >
>>
> func=xmlSecDSigCtxVerify:file=xmldsig.c:line=366:obj=unknown:subj=xmlSecDSigCtxSigantureProcessNode:error=1:xmlsec
>> > library function failed:
>> > Error: signature failed
>> > ERROR
>> >
>> > If I interpret it right, it seems like it's a problem of the underlying
>> > libxm2 library, but the question still stands. I have built the tool for
>> > x86_64 from the latest released source and used the latest libxml2 and
>> > libxslt sources as well.
>> >
>> > Thank you very much in advance.
>> > Vit Zikmund
>> >