Re: [g-a-devel] Questions about the gnome-mag design

[Bill Haneman <Bill Haneman Sun COM>]

Hi Carlos:

By the way, I did not mean to sound too negative in my reply.  Your 
questions are good ones, and I really thank you for your continued 
interest in gnome-mag.

> I agree with you that the separation of these two process are 
> important and I think that I really understand the  importance of 
> CORBA (or other IPC mechanism) to achieve this goal.

The desire to have the magnification "decoupled" from the client 
processing means we need either IPC or multi-threading.  My impression 
is that in a language like C, multi-threading is probably even harder to 
get right than IPC.

Also, there are situations where the magnifier service may need to run 
on a different host from the client.  For instance, if you are running a 
remote application, you don't really want the magnifier to run remotely 
as well, because of the much larger amount of X traffic and the CPU 
requirements of the magnification service.  CORBA allows this at least 
in theory (although our current 'bonobo-activation' mechanism does not 
recognize remote servers).  This sort of case is probably more common in 
some corporate environments, but we are not sure just how important it 
will be in the future.

> Just for information, what is scrolling?

Sometimes I used the words "scrolling" and "panning" interchangeably.  
The two primary tasks of a magnifier, I think, are creating a modified 
(rescaled) copy of a source image and keeping it up to date, and moving 
that rescaled image around on the target display, to keep it in the 
"right" position for the user.  The second task, moving a large image 
around with respect to a smaller viewport, is what I call "panning" or 
"scrolling".

It seems that from the user's point of view,  the most important thing 
is that this scrolling happens as rapidly and smoothly as possible.  If 
the scrolling is jerky or delayed, the user perceives that there is a 
performance problem with the magnification.  However, the user may be 
able to accept a delay of a few milliseconds in actually "updating" a 
region that has changed since it was last visible inside the viewport.  
This is why we separated the scrolling from the update processing inside 
gnome-mag, with different priorities.  This is also one of the 
complexities that can make the existing gnome-mag implementation code 
hard to understand and modify.

Technical note: On some other platforms this scrolling can be done 
faster and more smoothly for a number of technical reasons.  But the 
XWindow system doesn't really have a "scroll" command that we can use 
directly, all our scrolling has to be accomplished with XCopyArea and 
the generation of "expose" events.  Most of our scrolling requires that 
data be copied from the gnome-mag process memory to the XServer for all 
the newly-exposed areas, and we don't have a way to do this during the 
vertical blanking interval.  These are just some of the technical 
challenges facing the magnifier.  Some of the are made better with the 
COMPOSITE extension, and a magnifier based on the OpenGL APIs might also 
have an advantage, since it would probably be able to take advantage of 
hardware acceleration at least on some systems.

What we do now is "scroll" the existing client-side rescaled image into 
the viewport quickly, then process any resulting "updates" that need to 
happen in order to keep the rescaled image up-to-date with the source 
image.  We use the DAMAGE extension to get notification of when the 
source image has changed, and the XFIXES extension to get information 
about the onscreen mouse pointer so that we can rescale and re-display 
that as appropriate.

> > ...
> >
> > > What do you think in an effort to merge what BAUM developed in 
> > > gnopernicus with gnome-mag to create a complete solution for 
> > > magnification, eliminating these CORBA dependecies/complexities?
> >
> >
> >
> > This was explicitly avoided when gnopernicus, gnome-mag, and at-spi 
> > were initially designed, for many reasons.
>
>
> Could you list the more important/relevants reasons? They are these 
> ones that are still on the text, or have any others?

One reason is that gnopernicus is not, we hope, the only assistive 
technology to be developed for the free desktop, and we also hope that 
the existing 'sample implementation' of gnome-mag is not the only 
magnification service that will be written.  Already we have the 'orca' 
screen reader which probably will support magnification someday, but 
perhaps someone else will write a different client?  Some users want 
speech and magnification together, but some users may benefit from a 
smaller "magnification only" client.  Also, like orca, ATs might be 
written in languages other than C, which is another reason for using IDL 
because multiple language bindings are defined by the OMG standard.

So the existing design allows multiple clients to use the magnifier, 
separately or even at the same time, and it also allows clients like 
gnopernicus to choose which "gnome-mag:1.0" implementation to use.  For 
instance there are several ongoing efforts to write improved 
magnification services (for instance Gunnar Schmidt's is working on a 
COMPOSITE-based magnifier for school, and I believe there is at least 
one other effort to use COMPOSITE as a magnifier, and there is Keith 
Packard's experimental program called 'diopter').  If these programs 
were provided with gnome-mag IDL wrappers (basically just connecting the 
'impl' methods in the CORBA "skeletons"), then gnopernicus could choose 
to try different versions and select the best service for a given 
platform and user need.  The use of a common interface means that moving 
from one implementation to another is very straightforward.

> > > Why not design gnome-mag to be a library (or only a program like I 
> > > said in the last paragraph) that support magnification and export an 
> > > API in C? I have doubts if making this program an object that can be 
> > > accessed throw CORBA is the best solution.
> >
> >
> >
> > Creating a new set of client bindings with a C api may be reasonable, 
> > but there is already a C API - the one defined by the CORBA and IDL 
> > specification.  In any case this would need to wrap a client-server 
> > model, so some other IPC mechanism would have to be substituted for 
> > the CORBA one.
> > There are a number of areas where magnification can be improved, but 
> > I don't think that this one would give much useful improvement 
> > compared to the effort involved, and could be harmful.
> > Better improvements would include:
> >
> > * a multi-pass rendering system, so that better smoothing algorithms 
> > could be added to gnome-mag without slowing the panning performance 
> > unacceptably;
>
>
> Could you explain better what is a multi-pass rendering system or 
> point to some resources?

I described how we already have a sort of "multi pass" system, but it 
only makes two 'passes', it scrolls the existing scaled image into view, 
then updates any "stale" regions at idle time.  To make better use of 
smoothing algorithms, a more sophisticated 'multi pass' system would 
make more than one "pass" when rendering the updated image.  The first 
pass would be a fast, but possibly not high-quality, scaling algorithm 
like the existing GdkPixmap scaling operations we use.  Then, once all 
the pending "first pass" updates are done, a second pass at lower 
priority could begin, re-rendering the updated regions with a slower, 
higher-quality smoothing algorithm.  Some text smoothing algorithms are 
considerably slower than pixel replication or bilinear interpolation, 
but give less "blurry" results, so this would provide the magnification 
user with "quick feedback" when moving the mouse ("panning"), and "best 
quality" when looking at a static image, for instance when reading text.

The idea is similar to the idea behind JPEG images in a web browser that 
first display a blocky image, and then render the higher-quality image 
as the rest of the image data is downloaded.

best regards

Bill