Re: [xml] New libxml which is reentrant ?

[Roland Mainz <roland mainz nrubsig org>]

Daniel Veillard wrote:
> On Tue, Jul 08, 2008 at 09:57:09PM +0200, Roland Mainz wrote:
> > Daniel Veillard wrote:
> > > On Tue, Jul 08, 2008 at 02:04:32AM +0200, Roland Mainz wrote:
> > > > Are there any plans to make a new libxml version which is reentrant and
> > >
> > >   Define reentrant.
> >
> > The current problem we have is that if we have library "a" which uses
> > libxml2 and then calls a function in library "b" which itself does some
> > other stuff with libxml2 we get data corruption (the problem isn't
> > limited to one library - we're seeing this in multiple libxml2 consumers
> > in Solaris where libraries cannot be combined if both use libxml2 at the
> > same time).
>
>   Very strange. If that was true in general, that would mean your whole
> Gnome interface system would be unusable, as it uses many libraries and
> much of them rely on libxml2 for various things.

Erm... I think it's not the case since _noone_ has tried the following
stuff:
1. Start a new parsing session of a completely new document from within
a callback
2. Create a document from thread "a" and then pass it over to thread "b"
for further processing

AFAIK no consumer within Gnome or anything else in the OpenSolaris
codebase tries to do [1] or [2] ... ;-/

> I think this libxml2
> being used by many libraries is really a very common situation, not at
> all Solaris specific
> [root wei ~]# rpm -e libxml2 2>&1 | grep "is needed by" | wc -l
> 144
> this includes a lot of GNOME libraries for example, including GConf
> which in turn is used by any desktop app. Taking the example i'm
> most familiar with for virt-manager we use libxml2 heavilly both
> for libvirt virtualization level and for all the GNOME rendering.
> If what you're suggesting was right it would pop up everywhere...
>
>   By overgeneralizing, I'm afraid you discredit your case :-)

Right... my description wasn't very precise since we didn't had a full
diagnosis of the problem and I'm just back from a vacation and now try
to get to the bottom of this using dbx+Purify (in the meantime another
team has become more desperate (deadline approaching) and now work on
beefing-up "Parsify" to match their needs since it's fully reentrant).
The issue started when a completely unrelated library suddently used
libxml2 internally (without exposing the functionality to any of it's
own API) which is called from within a callback... and then all hell
broke loose (= SIGSEGV somewhere within |free()| and/or content from one
parsing session showing up in an unrelated one). The 2nd, overlapping
issue comes from using threads to create resources and then pass the
created resources to a different thread. And the 3rd issue is that some
libraries "override" the default memory allocators which becomes a
problem if another library or application does the same.

>   "Does some stuff" ... is that a description of an enginnering problem ?
> You may have an initialization problem. Possible that's described in a thread
> before. Is that related ? How is that linked to being 'reentrant' ?

With "reentrant" I was referring to an API which...
1. ... does not rely on global/thread-global data being modified, e.g.
stores _all_ read/write variables for a parsing seesion in a "handle"
object which is passed to all parser functions
2. ... allows "nesting", e.g. working on a different parsing session or
parser objects from within callbacks
3. ... allows moving objects between threads
4. ... does not use any non-reentrant functions itself

> What kind of processing is going on with libxml2 ?

Something is parsing a HTML document, calls a library function which
uses SAX to parse the application's preferences while another thread of
the application uses XmlReader. I'm try to get stacktraces larger this
week.

> What kind of corruption ?

Memory corruption and/or content from one parsing session "bleeding
through" into another parsing session.

----

Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland mainz nrubsig org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL <currently fluctuating>
 (;O/ \/ \O;)