SOLVED: AA pitfalls - might work for you! (mini-HOWTO)

["Idan Gazit" <id panix com>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hey everybody,

So I, too, was one of the unwashed masses suffering from non-working
AA after compiling garnome (0.12.2). My digging paid off and I would
like to share my success with you in the hopes that it cures your
problems as well. In my searching, I saw many of the same symptoms
that I had.

I'm writing this on the windows side of things until I get balsa to
compile, but feel free to email me if I've forgotten something.

So, a description of my setup is in order: 

 - 2.4.18 running on an intel box (laptop)
 - LFS - linux from scratch. A compiled-from-scratch 
   "distribution", no rpms, debs, or other associated nastiness.
 - the latest 2.x series stable gcc (2.95? I can check upon request.

   The important thing to note is that this is not RH's broken gcc 
   or gcc 3.x )
 - Xfree 4.2 (downloaded binaries -- this I didn't compile myself).

After some tweaking I got garnome to compile without a hitch.
Anxiously awaiting AA, I startx'ed. There appeared to be AA
statustext on the splash screen, but as soon as I got to the desktop
- -- nothing.

 * GDK_USE_XFT=1 was declared and exported correctly --
   not the culprit
 * the RENDER extension and xft were indeed loaded --
   not the culprit
 * freetype was installed correctly -- not the culprit
 * font paths and the like were correctly configured in 
   /etc/X11/XF86config and Xftconfig -- not the culprit
 * match edit antialias=true; set in Xftconfig -- go ahead and 
   put that in /etc/X11/Xftconfig somewhere. This causes all 
   fonts to be AA'd, at every point size. Once AA works, this 
   is tweakable, some people like AA at only smaller or larger
sizes.

Before you go digging for the deeper problems, make sure that the
basics are working. First off, I suggest some background reading:

   http://jmason.org/howto/subpixel.html

Despite the topic, almost everything there is applicable. Subpixel AA
is just an extra treat for those owning laptops or LCD monitors.
Unfortunately, it isn't up to date with gnome2 info. All the same, it
is an excellent resource for educating yourself on the ins and outs
of AA in X.

   http://www.linuxdoc.org/HOWTO/mini/FDU/index.html

The font deuglification howto has an informative section on AA. It
goes further in-depth about the config file for Xft.

The garnome README also has a blurb.

Read all three to become acquainted with the issues before
continuing.

The real culprit is at the bottom. If you're the type who likes to
peek at the last page of murder mysteries, go ahead. Note that I
suggest you read all of this before doing any of the steps, as
several of the steps require recompiling all or part of garnome and
you can save time by doing them all at once. Everything leading up to
it is a sort of log of my journeys to get AA working, and something
of a loose summary of the stuff you can get from searching about the
net.

 * in your .xinitrc, add a line at the top saying "export 
   GDK_USE_XFT=1". If you've done this right, you can check 
   for its presence by opening up a gnome-term and typing 
   "printenv" (or shell-specific equivalent). GDK_USE_XFT=1 
   should show up somewhere in there. If you login via gdm 
   then this should probably go elsewhere though I don't 
   know where. This shouldn't be difficult to find out 
   though...

 * look through xfree's log for the RENDER extension. A nice 
   way of doing this is posted on Justin Mason's excellent page:

   xdpyinfo | grep -c -i render

   If you don't get "1" as the response, your problem is X. 
   Reinstall xfree86 and make sure that you have the render 
   extension and all of the dev includes. As a rabid denouncer 
   of everything RPM, I suggest getting the binaries from xfree 
   along with their nice installer script. I use it, and it 
   works for me.

 * make sure you have a new version of freetype installed. 
   Although I can't be sure, I "broke" garnome's order and 
   installed freetype first. As a library, it relies on nothing. 
   This probably had no real effect on things, but it might
   have. 

 * make sure your font paths and the like are correct in xfree's 
   config files. There are endless explanations on the net on how 
   to make fonts available to X, suffice it to say that if they 
   are not you won't have anything interesting to AA.

 * download yourself microsoft's nice TT fonts. Kieth Packard has
   kindly prepared a tarball all set up for immediate use:

   http://keithp.com/~keithp/truetype.tar.gz

   download and unpack to your /usr/X11R6/lib/X11/fonts dir (?).
   Name the dir something convenient (say, "truetype") and add
   it to your fontpath (one entry in /etc/X11/XF86config, another
   in /etc/X11/Xftconfig). 

 * add the line:

   match edit antialias=true;

   to your /etc/X11/Xftconfig.


Assuming all of the above are correct, execute the following
command:

xterm -fa "Andale Mono"

It should produce an xterm with lovely smooth Andale Mono. If you
don't have Andale Mono, try another font that you do have. If it
still doesn't work, something is wrong with X or freetype or your
fontpaths or your Xftconfig. Go back and check the previous steps. 

Really. If the above doesn't work you have deeper problems that
nothing ahead will fix.

OK. If you are here you can get AA through the xterm, but for some
unknown reason it doesn't work anywhere else.

A diversion on how AA works in gnome2: pango (the new text-rendering
backend) is responsible for smoothing your fonts and gtk+2 is
responsible for drawing them (this is my very loose understanding,
and also what helped me to crack this problem). Pango, in turn, is
dependent on freetype to render individual glyphs.

PART I: paths and ld.so.conf

Chances are, you're getting screwed here, especially if you're
installing alongside gnome 1.4 (I wasn't, but this was still a
problem).

Basically, wherever garnome goes must always be ahead in your path.
The reason for this is that various elements of gnome find out how to
compile themselves by running a small app for each of their
dependencies to find out where their libraries and includes lie. For
example: foo is dependant on bar. Bar has already been installed in
/opt/gnome2/, and now we are compiling foo. Bar has left a small
application called "bar-config" in /opt/gnome2/bin -- and now
anything that is dependent on bar can run bar-config to find out
where its libraries lie.

Where is the pitfall? Say we have an older version of bar in
/opt/gnome1/. If /opt/gnome1/bin is ahead of /opt/gnome2/bin in the
path, foo will compile against the gnome1 version of bar. If it
compiles at all, this situation creates a great deal of unhappiness.

The same problem exists for your /etc/ld.so.conf -- make sure this is
set so that your gnome2 libs are ahead of anything involving gnome1.

An important pitfall here is that some distros may put some of the
gnome1 pieces -- say freetype -- in /usr/ or /usr/local/. You *MUST
MAKE SURE* that anything compiling in garnome sees the other garnome
pieces before it sees the older crud from your nasty distro. Put the
build dir (/opt/gnome2 or what have you) first in everything that
resembles a path. Recompile garnome.

I got screwed by this in the start and had to recompile garnome. On
to...

PART II: Xfree and Freetype

Apparently, freetype came with my binary install of Xfree86, and of
course, it was quite stale. The README in freetype's tarball mentions
this, and mentions how to remedy the situation. Basically, you've got
to replace the libraries (I believe they lived in /usr/X11R6/lib).
You should have three files:

libfreetype.so      \
libfreetype.so.6    | } or something similar. I can't remember if
the
libfreetype.so.6.2  /   version comes before or after the .so

Note that the top two are symlinks and only the bottom is the actual
library. Get rid of all three. Make new symlinks with the same names
as the old ones, but which point to the new freetype library you've
installed with garnome. Make sure they point to the actual library,
and not to other symlinks. The final result is that you should have
your libfreetype.so and libfreetype.so.6 pointing at your brand-new
freetype, /opt/gnome2/lib/libfreetype.so.6.3 or what have you. 

Become root. Run "ldconfig". This may or may not be related to the
problem, but I did it, so I impart my knowledge to you. Move on
to...

PART III: The real culprit

OK, no more suspense. What ruined my AA is the fact that pango was
unable to find the Xft includes in /usr/X11R6/include/X11/Xft/. Thus,
pango did not compile the xft backend, and thus no AA.

How can you check for this? Become root. Type "ldconfig -vv | less".
Look through it for a reference to "libpangoxft". Nothing? This is
what got your goat.

Go back to your garnome directory and cd to
gnome/pango/work/pango.<someversion>. Look through the config.log for
mentions of xft. If any of them show up as no, this is what caused
pango not to compile libpangoxft. My compile failed here because
/usr/X11R6/include wasn't in the include paths. 

My solution was to export a CPPFLAGS that did include it, and remake
pango:

export CPPFLAGS=-I/usr/X11R6/include/

Change to the <garnome>/gnome/pango dir (the one with the Makefile
and checksum). Get rid of downloads/, cookies/, and work/. "make
install". Once everything is finished, re-run ldconfig as mentioned
above and look through it for libpangoxft. If you've succeeded,
great! One thing left to do...

GTK is dependent on pango, so you need to recompile it as well.
Fortunately, all you really need to do is to change to your
<garnome>/gnome/gtk+ dir and get rid of everything as with pango, and
"make install". Run ldconfig once for good measure. Restart X.

Hopefully you are now looking at AA text.

CONCLUSION

I hope this guide has helped you. It was cobbled together from two
parts sleep deprivation, one part determination, fifteen parts
exasperation, and one fat helping of infatuation with AA text, which
is what motivated me not to give this fight up. I know it isn't the
clearest of guides, and nothing here may help, but perhaps what
killed me got you as well, and I wish somebody had written this for
me. All of the guides out there seem to deal with KDE, or gdkxft (the
gnome 1.2 AA patch/hack). All of them mention that "sometime in the
future" gnome 2.0 will have AA by default, but stop there. As garnome
and gnome2 evolves and settles in, perhaps I will hack this text into
some semblance of a proper howto, if enough interest is shown. 

Direct your spam and flames to /dev/null. Direct constructive
criticism, praise, money, requests, questions, etc. to <id /at/ panix
/dot/ com>. Especially money. (Students are poor).

Cheers,

Idan

- --
id at panix dot com
PGP key id 0x60c030d2

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (MingW32) - WinPT 0.5.12
Comment: For info see http://www.gnupg.org

iD8DBQE9Myp2Ilf5wmDAMNIRApJSAKD8yXVsEaDfdIbPEX8ZT1JdBXHh5gCeLmVv
EBZ/y99aABGKi53C60wiicA=
=/+Rk
-----END PGP SIGNATURE-----