[xmlsec] Support for really large XML documents

[Vit Zikmund]

Hi Aleksey, thanks for the tip.
I've tried it, but apparently, it's not the case. I've debugged the code 
and found the source of the error.
Here http://git.gnome.org/browse/xmlsec/tree/src/c14n.c#n277 
xmlOutputBufferClose(buf) returns negative number, but it's not an error 
code - it's an overflowed byte counter.
The overflow happens without error during the transformation execution in 
the libxml2 code - at the end of xmlOutputBufferWrite() ( 
http://git.gnome.org/browse/libxml2/tree/xmlIO.c#n3445 ).
Everything is just an 'int' over there. If I add a line checking for 
overflow to keep the value positive, my test passes, but that is some 
nasty hack.

I've already contacted the author and he said such big value shouldn't 
ever be there and suggested this might be a bad design.
This is the thread on libxml mailing list: 
https://mail.gnome.org/archives/xml/2012-May/msg00075.html

Can you comment on that? Might this be related to your comment few lines 
above the error saying:
/* we are using a semi-hack here: we know that xmlSecPtrList keeps
 * all pointers in the big array */

Thanks again,
Vit

Might this be somehow related to the comment few lines above

Aleksey Sanin <aleksey at aleksey.com> wrote on 05/23/2012 09:28:11 PM:

> The error indicates that we can't allocate output buffer correctly. If
> I would guess, then I would see if the "size" parameter is treated as
> negative number when it exceeds 2G.
> 
> Try to change include/xmlsec/xmlsec.h and change the xmlSecSize to be
> a typedef to size_t all the time (dont' forget to recompile xmlsec
> after this change).
> 
> Aleksey
> 
> On 5/23/12 8:28 AM, Vit Zikmund wrote:
> > Hello,
> > we are trying to use the XMLSec utility to verify documents signed 
with
> > our own application and probably have hit a limit of the document 
size,
> > that XMLSec is able to process.
> > 
> > The simplest question is: Does XMLSec support handling large
> > documents/files? Is is about to? For large I mean 2 gigabytes and 
more.
> > 
> > I can verify a document of 1GB, but little over 2GB returns an error:
> > 
> > 
> 
func=xmlSecTransformC14NPushXml:file=c14n.c:line=279:obj=c14n:subj=xmlOutputBufferClose:error=5:libxml2
> > library function failed:
> > 
> 
func=xmlSecTransformDefaultPushXml:file=transforms.c:line=2417:obj=enveloped-
> signature:subj=xmlSecTransformPushXml:error=1:xmlsec
> > library function failed:
> > 
> 
func=xmlSecTransformCtxXmlExecute:file=transforms.c:line=1236:obj=unknown:subj=xmlSecTransformPushXml:error=1:xmlsec
> > library function failed:transform=enveloped-signature
> > 
> 
func=xmlSecTransformCtxExecute:file=transforms.c:line=1296:obj=unknown:subj=xmlSecTransformCtxXmlExecute:error=1:xmlsec
> > library function failed:
> > 
> 
func=xmlSecDSigReferenceCtxProcessNode:file=xmldsig.c:line=1571:obj=unknown:subj=xmlSecTransformCtxExecute:error=1:xmlsec
> > library function failed:
> > 
> 
func=xmlSecDSigCtxProcessSignedInfoNode:file=xmldsig.c:line=804:obj=unknown:subj=xmlSecDSigReferenceCtxProcessNode:error=1:xmlsec
> > library function failed:node=Reference
> > 
> 
func=xmlSecDSigCtxProcessSignatureNode:file=xmldsig.c:line=547:obj=unknown:subj=xmlSecDSigCtxProcessSignedInfoNode:error=1:xmlsec
> > library function failed:
> > 
> 
func=xmlSecDSigCtxVerify:file=xmldsig.c:line=366:obj=unknown:subj=xmlSecDSigCtxSigantureProcessNode:error=1:xmlsec
> > library function failed:
> > Error: signature failed
> > ERROR
> > 
> > If I interpret it right, it seems like it's a problem of the 
underlying
> > libxm2 library, but the question still stands. I have built the tool 
for
> > x86_64 from the latest released source and used the latest libxml2 and
> > libxslt sources as well.
> > 
> > Thank you very much in advance.
> > Vit Zikmund
> > 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.aleksey.com/pipermail/xmlsec/attachments/20120524/da279892/attachment.html>