Re: gnome-vfs/GIOChannel for parsing

[Jody Goldberg <jody gnome org>]

On Wed, Mar 05, 2003 at 09:35:25PM -0500, Colin Walters wrote:
> What do you use Storages for?  How is it different from just using a
> particular stream implementation?
> 
Storages contain streams in MS parlance.  In broad strokes there are
two forms of storage.
    - structured files : The containing stream has a format which
      defines sub streams.  Each sub-stream can be read individually.
      Things like tar, ar, and if you ignore compression zip.

    - file system in a file : An extension to structured files that
      allows simultaneous reading/reading from sub-streams.
      Equivalent to being able to open a sub-stream and write data
      to it, having the implementation magically handle extending
      the meta-data/allocation-tables/... to manage the content.

Interfaces and implementations for these formats is important for
applications as soon as we start combining multiple applications.
If I want Gnumeric to contain some drawing managed by dia or
sodipodi it is inconvenient to store all of them in the same xml
blob.  Nor is it pleasant to store binary data (images sound) as
base64 encoded text in the xml.  We need to support structured files
at a minimum.  It is an open question whether to extend things and
support file system in a file is necessary.  I am leaning towards
no (which is why gsf is as it is) but there are some compelling
arguments in favour.
 
There are several other design issues that could be discussed
before things go into glib.

> > a simple assumption and added a GsfInput::size method which assumed
> > that we could always know the size of the data to be read when it
> > was opened.  This is not true for bzip2 files.  Its this sort of
> > thing that needs to be hashed out now.
> 
> Fair enough.  In that case though you could just make the size method
> return -1 or something if the size isn't known.

We considered this and rejected the notion because it would break
several use cases (many already in existence).  
 
> > I am talking about merging parts of gsf into glib.  I envision
> > something on the order of gdk pixbuf.  With a registry of known
> > formats and interface specifications in glib.  Anything less seems
> > pointless.
> 
> I'm not so sure about the registry, but OK.

Having an abstract interface in glibs is clearly a good idea, along
with several implementations for the basic operations.  However, I
suspect that people would not want MS OLE2 parsers in glib proper.

An important use of the stream interface would be to support the
mime sniffing implementation being discussed else where.  They had
better work together or they are pointless.  So you need to be able
to sniff the type of a stream (with or without knowing its name) and
potentially guess a type knowing only a name.  To support proper
content based sniffing and intelligent mime type support it would be
nice to support gdk-pixbuf style operations.  Those depend on an
external registry of known types that can install themselves without
significant startup cost to most alps. 

> What do you mean by "convenience wrapper"?  In the Mono .NET
> implementation they just cheated and had the default implementations in
> Stream call the blocking versions.   We could similarly have the API
> there, but not implement it fully until later.

I'll try to be more explicit.  My meaning is that an api to support
asynchronous operations is orthogonal to the stream interfaces.
These are very central concepts and there is no rush.  Lets pick
pieces and work on them.  I'm willing to use gsf as a test bed
because there are no stable projects depending on them.  However,
that will change soon.

As a starting point would you like to work on some TextReader &
StreamReader interfaces that wrap a GsfInput (String Reader seems
pointless given GsfInputMemory an C strings).  The basic interface
of TextReader seems similar to my intent with GsfInputTextline.

I don't see alot of use to some of the TextReader interfaces
    - get/initialize LifetimeService
    - hashing
    - equality
    - serializable
    - get as string (seems like a convenience routine on the stream)

On further examination of the .net Stream interface it seems
fundamentally different from my goals in gsf.  Its base 'Stream'
interface mixes in and out.  There is no way to divide them.  I
would be very much against using that as the primary abstracting in
glib.