[xmlsec] Support for really large XML documents

[Vit Zikmund]

I don't blame you. That's perfectly fine with me. However, how do you 
think it should be fixed?

Vit

Aleksey Sanin <aleksey at aleksey.com> wrote on 05/24/2012 08:11:54 PM:

> Unfortunately, I have to have the whole document in memory for C14N
> 
> Aleksey
> 
> On 5/24/12 11:08 AM, Vit Zikmund wrote:
> > Hi Aleksey, thanks for the tip.
> > I've tried it, but apparently, it's not the case. I've debugged the 
code
> > and found the source of the error.
> > Here
> > _http://git.gnome.org/browse/xmlsec/tree/src/
> c14n.c#n277_xmlOutputBufferClose(buf)returns
> > negative number, but it's not an error code - it's an overflowed byte
> > counter.
> > The overflow happens without error during the transformation execution
> > in the libxml2 code - at the end of *xmlOutputBufferWrite*() (
> > _http://git.gnome.org/browse/libxml2/tree/xmlIO.c#n3445_).
> > Everything is just an 'int' over there. If I add a line checking for
> > overflow to keep the value positive, my test passes, but that is some
> > nasty hack.
> > 
> > I've already contacted the author and he said such big value shouldn't
> > ever be there and suggested this might be a bad design.
> > This is the thread on libxml mailing list:
> > _https://mail.gnome.org/archives/xml/2012-May/msg00075.html_
> > 
> > Can you comment on that? Might this be related to your comment few 
lines
> > above the error saying:
> > /* we are using a _semi_-hack here: we know that xmlSecPtrList keeps
> >  * all pointers in the big array */
> > 
> > Thanks again,
> > Vit
> > 
> > Might this be somehow related to the comment few lines above
> > 
> > Aleksey Sanin <aleksey at aleksey.com> wrote on 05/23/2012 09:28:11 PM:
> > 
> >> The error indicates that we can't allocate output buffer correctly. 
If
> >> I would guess, then I would see if the "size" parameter is treated as
> >> negative number when it exceeds 2G.
> >>
> >> Try to change include/xmlsec/xmlsec.h and change the xmlSecSize to be
> >> a typedef to size_t all the time (dont' forget to recompile xmlsec
> >> after this change).
> >>
> >> Aleksey
> >>
> >> On 5/23/12 8:28 AM, Vit Zikmund wrote:
> >> > Hello,
> >> > we are trying to use the XMLSec utility to verify documents signed 
with
> >> > our own application and probably have hit a limit of the document 
size,
> >> > that XMLSec is able to process.
> >> >
> >> > The simplest question is: Does XMLSec support handling large
> >> > documents/files? Is is about to? For large I mean 2 gigabytes and 
more.
> >> >
> >> > I can verify a document of 1GB, but little over 2GB returns an 
error:
> >> >
> >> >
> >>
> > 
> 
func=xmlSecTransformC14NPushXml:file=c14n.c:line=279:obj=c14n:subj=xmlOutputBufferClose:error=5:libxml2
> >> > library function failed:
> >> >
> >>
> > 
> 
func=xmlSecTransformDefaultPushXml:file=transforms.c:line=2417:obj=enveloped-
> >> signature:subj=xmlSecTransformPushXml:error=1:xmlsec
> >> > library function failed:
> >> >
> >>
> > 
> 
func=xmlSecTransformCtxXmlExecute:file=transforms.c:line=1236:obj=unknown:subj=xmlSecTransformPushXml:error=1:xmlsec
> >> > library function failed:transform=enveloped-signature
> >> >
> >>
> > 
> 
func=xmlSecTransformCtxExecute:file=transforms.c:line=1296:obj=unknown:subj=xmlSecTransformCtxXmlExecute:error=1:xmlsec
> >> > library function failed:
> >> >
> >>
> > 
> 
func=xmlSecDSigReferenceCtxProcessNode:file=xmldsig.c:line=1571:obj=unknown:subj=xmlSecTransformCtxExecute:error=1:xmlsec
> >> > library function failed:
> >> >
> >>
> > 
> 
func=xmlSecDSigCtxProcessSignedInfoNode:file=xmldsig.c:line=804:obj=unknown:subj=xmlSecDSigReferenceCtxProcessNode:error=1:xmlsec
> >> > library function failed:node=Reference
> >> >
> >>
> > 
> 
func=xmlSecDSigCtxProcessSignatureNode:file=xmldsig.c:line=547:obj=unknown:subj=xmlSecDSigCtxProcessSignedInfoNode:error=1:xmlsec
> >> > library function failed:
> >> >
> >>
> > 
> 
func=xmlSecDSigCtxVerify:file=xmldsig.c:line=366:obj=unknown:subj=xmlSecDSigCtxSigantureProcessNode:error=1:xmlsec
> >> > library function failed:
> >> > Error: signature failed
> >> > ERROR
> >> >
> >> > If I interpret it right, it seems like it's a problem of the 
underlying
> >> > libxm2 library, but the question still stands. I have built the 
tool for
> >> > x86_64 from the latest released source and used the latest libxml2 
and
> >> > libxslt sources as well.
> >> >
> >> > Thank you very much in advance.
> >> > Vit Zikmund
> >> >
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.aleksey.com/pipermail/xmlsec/attachments/20120524/72501882/attachment-0001.html>