[xmlsec] Support for really large XML documents

[Aleksey Sanin]

In the ideal world, the xml buffer functions should use size_t instead
of int.

Sorry, I don't think there is an easy fix

Aleksey

On 5/24/12 11:20 AM, Vit Zikmund wrote:
> I don't blame you. That's perfectly fine with me. However, how do you
> think it should be fixed?
> 
> Vit
> 
> Aleksey Sanin <aleksey at aleksey.com> wrote on 05/24/2012 08:11:54 PM:
> 
>> Unfortunately, I have to have the whole document in memory for C14N
>>
>> Aleksey
>>
>> On 5/24/12 11:08 AM, Vit Zikmund wrote:
>> > Hi Aleksey, thanks for the tip.
>> > I've tried it, but apparently, it's not the case. I've debugged the code
>> > and found the source of the error.
>> > Here
>> > _http://git.gnome.org/browse/xmlsec/tree/src/
>> c14n.c#n277_xmlOutputBufferClose(buf)returns
>> > negative number, but it's not an error code - it's an overflowed byte
>> > counter.
>> > The overflow happens without error during the transformation execution
>> > in the libxml2 code - at the end of *xmlOutputBufferWrite*() (
>> > _http://git.gnome.org/browse/libxml2/tree/xmlIO.c#n3445_).
>> > Everything is just an 'int' over there. If I add a line checking for
>> > overflow to keep the value positive, my test passes, but that is some
>> > nasty hack.
>> >
>> > I've already contacted the author and he said such big value shouldn't
>> > ever be there and suggested this might be a bad design.
>> > This is the thread on libxml mailing list:
>> > _https://mail.gnome.org/archives/xml/2012-May/msg00075.html_
>> >
>> > Can you comment on that? Might this be related to your comment few lines
>> > above the error saying:
>> > /* we are using a _semi_-hack here: we know that xmlSecPtrList keeps
>> >  * all pointers in the big array */
>> >
>> > Thanks again,
>> > Vit
>> >
>> > Might this be somehow related to the comment few lines above
>> >
>> > Aleksey Sanin <aleksey at aleksey.com> wrote on 05/23/2012 09:28:11 PM:
>> >
>> >> The error indicates that we can't allocate output buffer correctly. If
>> >> I would guess, then I would see if the "size" parameter is treated as
>> >> negative number when it exceeds 2G.
>> >>
>> >> Try to change include/xmlsec/xmlsec.h and change the xmlSecSize to be
>> >> a typedef to size_t all the time (dont' forget to recompile xmlsec
>> >> after this change).
>> >>
>> >> Aleksey
>> >>
>> >> On 5/23/12 8:28 AM, Vit Zikmund wrote:
>> >> > Hello,
>> >> > we are trying to use the XMLSec utility to verify documents
> signed with
>> >> > our own application and probably have hit a limit of the document
> size,
>> >> > that XMLSec is able to process.
>> >> >
>> >> > The simplest question is: Does XMLSec support handling large
>> >> > documents/files? Is is about to? For large I mean 2 gigabytes and
> more.
>> >> >
>> >> > I can verify a document of 1GB, but little over 2GB returns an error:
>> >> >
>> >> >
>> >>
>> >
>>
> func=xmlSecTransformC14NPushXml:file=c14n.c:line=279:obj=c14n:subj=xmlOutputBufferClose:error=5:libxml2
>> >> > library function failed:
>> >> >
>> >>
>> >
>>
> func=xmlSecTransformDefaultPushXml:file=transforms.c:line=2417:obj=enveloped-
>> >> signature:subj=xmlSecTransformPushXml:error=1:xmlsec
>> >> > library function failed:
>> >> >
>> >>
>> >
>>
> func=xmlSecTransformCtxXmlExecute:file=transforms.c:line=1236:obj=unknown:subj=xmlSecTransformPushXml:error=1:xmlsec
>> >> > library function failed:transform=enveloped-signature
>> >> >
>> >>
>> >
>>
> func=xmlSecTransformCtxExecute:file=transforms.c:line=1296:obj=unknown:subj=xmlSecTransformCtxXmlExecute:error=1:xmlsec
>> >> > library function failed:
>> >> >
>> >>
>> >
>>
> func=xmlSecDSigReferenceCtxProcessNode:file=xmldsig.c:line=1571:obj=unknown:subj=xmlSecTransformCtxExecute:error=1:xmlsec
>> >> > library function failed:
>> >> >
>> >>
>> >
>>
> func=xmlSecDSigCtxProcessSignedInfoNode:file=xmldsig.c:line=804:obj=unknown:subj=xmlSecDSigReferenceCtxProcessNode:error=1:xmlsec
>> >> > library function failed:node=Reference
>> >> >
>> >>
>> >
>>
> func=xmlSecDSigCtxProcessSignatureNode:file=xmldsig.c:line=547:obj=unknown:subj=xmlSecDSigCtxProcessSignedInfoNode:error=1:xmlsec
>> >> > library function failed:
>> >> >
>> >>
>> >
>>
> func=xmlSecDSigCtxVerify:file=xmldsig.c:line=366:obj=unknown:subj=xmlSecDSigCtxSigantureProcessNode:error=1:xmlsec
>> >> > library function failed:
>> >> > Error: signature failed
>> >> > ERROR
>> >> >
>> >> > If I interpret it right, it seems like it's a problem of the
> underlying
>> >> > libxm2 library, but the question still stands. I have built the
> tool for
>> >> > x86_64 from the latest released source and used the latest
> libxml2 and
>> >> > libxslt sources as well.
>> >> >
>> >> > Thank you very much in advance.
>> >> > Vit Zikmund
>> >> >
>>