Re: A GIterator type, but not as a simple type -> as a GTypeInterface (GIterIface)

[Owen Taylor <otaylor redhat com>]

On Wed, 2006-02-08 at 01:44 +0100, Philip Van Hoof wrote:
> Hey Owen,
> 
> More then a year ago you wrote an E-mail about why you wouldn't want to
> start using a type like GIterator in gtk+.
> 
> I do share some of your opinions on this, mainly the fact that it
> shouldn't be introduced in gtk+. 
> 
> But I wouldn't share your opinion if you would say that therefore a type
> like GIterator shouldn't be available for developers that use GObject.

I'll stick to my opinion that adding GIterator is something you'd
want to do only if the amount of additional complexity you were
going to build on top of GLib and GObject in the future far 
exceeds the amount of complexity that exists today.

And the thought of the possibility of that happening makes me sad :-(

> I wrote a reply to your E-mail.
> 
> 
> > Wanted to definitely WONTFIX 
> > 
> >  http://bugzilla.gnome.org/show_bug.cgi?id=83729
> > 
> > about GIterator, which has been in limbo for a long time, so I found a
> > writeup I started a year or two ago and finished it off.
> > 
> > The basic question to me is "what is GIterator trying to achieve?".
> > 
> > Possibilities:
> > 
> >  1) Consistency. We have many ways of returning lists of
> >     items or iterating over sets of items. Conceivably,
> >     by consitently always using GIterator in these 
> >     circumstances, we could have a single way of doing
> >     everything.
> > 
> >     To me, this is a lost battle. GTK+ is relatively mature 
> >     codebase - is GTK+ going to have twice as many entry points 
> >     10 years from now? I doubt it. So, if we only used
> >     GIterator for new additions, we'd still be > 50% 
> >     inconstistent.
> >
> 
> I don't think a type like GIterIface should become heavily used by the
> gtk+ library. That doesn't mean it's not necessary. to have such a type.
> People use GObject, and it's possible, to create object oriented cross
> programming platform reusable libraries. They typically generate
> language bindings for it. There's multiple problems with the current
> ways of returning lists (like returning a GList). One of the problems is
> (as you noted here) knowing the type.
> 
> An iterator as an interface can improve this situation. Specialized
> type-correct methods could be made public in a GObject that implements a
> type-generic iterator interface. A language binding can be instructed to
> use only the type-correct signatures.

This can be solved with much less overall complexity in the framework
of the introspection work that has been started. 

> In stead of the original proposals that made it possible to inject
> function pointers to handle with list-types, an interface simply does it
> how programming environments like Java and .NET do it: let the developer
> of the list-type implement an iterator implementation for it.

So, every time we want to return a list, we have to create another
GObject type for the return value? Even with G_DEFINE_TYPE(), that's
not a very appealing prospect.

Implementing an iterator in Java or C# is just a few lines of code
because type definition is native, because there is garbage collection,
so you don't spend 3/4's of your code managing memory.

> >     And you can't get consistency by adding extra function 
> >     variants ... the old ones will still be there in the
> >     docs, still be there in old code.
> >   
> >     In my opinion, adding GIterator is not going to make
> >     it easier for people to learn how to program GTK+ in C.
> 
> Right. And it perhaps shouldn't be used in Gtk+. Perhaps except as a
> replacement for GtkTreeIter. While developing custom GtkTreeModel
> implementations, I more then once wanted to have more power over that
> GtkTreeIter thingy. Right now I need to implement the "next" method in
> my GtkTreeModel implementation. This is typically something you would
> implement in your iterator implementation, not in the list-model.

Avoiding having to memory manage the iterators was a goal for
GtkTreeView development. That makes C coding with GtkTreeView
distinctly easier. But also, do some profiling of GObject creation
speed. It's pretty slow.

> This way it becomes impossible to reuse your iterator implementation.
> You need to re-implement it in the list-model implementation. Which
> basically leads to code duplication when developing custom tree models.
> Code duplication leads to bugs and hard-to-understand code. 

If you actually have cases where you can use the same data structures
for many tree models (the whole idea of writing many tree models
is a weird to me), then you should be able to share the code without
much difficulty, maybe with a little boilerplate in your GtkTreeModel
code, but without duplication.

> What if in four years ten new developers join my team. Will I have to
> explain all the many quirks I had to implement because gtk+ didn't allow
> me to use my own iterators? This will cost my company a lot money.

Why don't you save your company a lot of money and get your developers
working in a more pleasant language than C?

It definitely bothers me that it's hard to write GtkTreeModel
implementations in a high-level language. That's something I wish we
could have done better. But I don't know how, and I'm not entirely
sure it was possible ... the memory management paradigms we use
for GTK+ work well for big objects, they don't work well for tiny
objects.

I almost wonder whether the right way to attack things is to provide
only two things:

 - A view that is *much* more of a shell than our current view -
   it just has callbacks for drawing.
 - An interface spec for what the final result should look like
   with example mappings to various languages.

And then force language binding authors to do most of the heavy lifting
on their side. That's painful, but you can get some pretty big wins
in this area by being pure Java or pure CLR.

Then again, most people don't really need to do a custom model at all;
it's almost always a mistake.

> It would be more easy if I could simply implement the iterator, create
> unit tests for it, and tell the developers something like: look, THIS is
> what you use to iterate over THAT list-type.
> 
> Now I have to tell them: you use THIS for iterating over THAT list-type,
> except when you are developing a custom tree model. In THAT case you
> need to do gtk+ hocus pocus. Because, well .. some gtk+ developers think
> iterators aren't worth it.

If someone is developing custom tree models in C, they better be
comfortable enough with GLib and C not to wonder why they can't use
GList functions on a GtkTreeModel....

> Hrmmmmm :-)

> >  2) Better memory management. One problem with the current
> >     methods of returning lists or arrays is that the
> >     caller has to figure out how to free the result.
> >     GIterator standardizes that.
> 
> Haven't thought about that one. Okay.
>  
> >  3) Language binding support. Array and list returns are
> >     currently a problem for language bindings because
> >     of the memory management concerns in 2). foreach()
> >     functions don't have this problem, but they typically
> >     require some hand language binding glue and may not
> >     be very natural for some programming languages. (For 
> >     instance, C++) 
> 
> >     A second problem for language bindings is that there
> >     is missing type information if we just have a 
> >     function signature returning a GSList.
> 
> Above I explained how doing the iterator type as an interface can solve
> this.

I was discussing a particular proposal, that was GLib level
not GObject level. If you do iterators as a GObject thing that returns
GValues, you've made things even more unpleasant for C programmers, but
yes, you'll solve this problem.

> >     GIterator and GValueIterator do help the memory 
> >     management problem .. see above.
> > 
> >     GIterator doesn't solve the type information problem.
> 
>    
> >  4) Convenience. Maybe it is just easier to use GIterator
> >     than the current APIs. I don't see this as the case ...
> >     it's more function calls. It's not type safe. (Some
> >     of the current calls aren't, some are.) You can't
> >     make multiple passes through the data in an easy
> >     fashion.
> 
> Type safety can be implemented if you do the iterator type as an
> interface.

Did generics get added to C when I wasn't watching?

> > None of these is all that compelling to me.
> > 
> > One other thing to note is that with iterators you have
> > to choose between
> > 
> >  - Lack of robustness like a foreach() function; if iterators
> >    contain pointers to internal data, they won't be 
> >    robust against structural changes.
> 
> Oh? Can you give an example?

Your iterator contains a pointer to a GList node. While your 
iterator is kept around, the GList node is freed...

> >  - Lack of efficiency like returning an array. You can 
> >    make the iterator robust by just making it a wrapper
> >    around an array.
> 
> Efficiency is important for the core gtk+ libraries (internal gtk+
> code). I don't think 90% of the desktop application developers are
> interested in such levels of efficiency. Anyway, you can let the
> implementer of a GIterIface for a specific type implement efficient
> foreach, next, previous and has_next methods and properties. The
> efficiency is now in the hands of the GIterIface implementer, it's no
> longer in the hands of the desktop application developer. This is in
> many cases going to lead to more efficient application code.

I don't think there is any magic efficiency sauce here.... if you
want robustness guarantees you have to copy.

(Note that Java, for example, doesn't guarantee safeness against
modifications during iterations. But garbage collection means
you can frequently get away with more than you can in C ... 
if the iterator references a collection element, it isn't
going to vanish.)

> > The fact that we can't achieve consistency is the killer objection 
> > to me. Memory management does get better, language binding
> > support and convenience is a bit of a wash. But without
> > being able to use GIterator consitently everywhere, in the
> > end all we will be making it harder to learn how to use 
> > GLib and GTK+, not easier.
> 
> 
> That way, you block 95% of all innovation, Owen ;-).

You can't just keep piling the complexity higher and deeper and
call it innovation.

> I'd say implement a library on top of GObject that makes it possible to
> start using a type like GIterIface. But not yet use it in gtk+. Perhaps
> refactoring the GtkTreeView so that letting it use a real iterator in
> stead of GtkTreeIter. I would in stead of putting a new
> GtkTreeModelIface in gtk+, put it in that library on top of GObject. For
> example with a name like GListModelIface and GTreeModelIface.

I think we've just about got people over the GtkCList => GtkTreeView
hump now... not particularly looking forward to another switch.

Regards,
						Owen