Announcing Dolt, a drop-in Libtool replacement which cuts build times in half

[Josh Triplett <josh freedesktop org>]

Many packages use GNU autotools (automake and autoconf) to build, to
the point that "./configure && make" represents one of the most common
build procedures for Free Software packages.  Libraries using
autotools typically use GNU Libtool, partly because it works on almost
any system and partly because autotools makes it difficult to do
otherwise.  Packages which use these libraries sometimes use libtool
as well.

Yet for many of these libraries and other packages, more than half of
the build time goes into running the libtool shell script.

Libtool knows how to handle libraries for umpteen different systems,
including many ancient systems that have terrible shared library
support.  It has some extensive shell script logic to figure out how
to build libraries for your system, and how to compile objects that go
in those libraries.  This logic does an amazingly impressive job of
coping with adverse conditions.  However, this logic all lives in an
~8500 line, ~250kB shell script, which runs *every single time you
compile a source file*.

This does not do wonders for performance.

Meanwhile, modern systems such as GNU/Linux have reasonable library
mechanisms, and need relatively little of the machinery in libtool.
On these common systems, it would significantly improve build times to
avoid running that libtool machinery for every compilation.

Thus, I wrote Dolt, a drop-in replacement for libtool's compilation
mode.  Dolt runs any necessary system-specific or
configuration-specific logic as part of configure, writes out a simple
shell script "doltcompile"[1], and substitutes it for libtool in the
automake variables LTCOMPILE and LTCXXCOMPILE.  If you use automake,
autoconf, and libtool, then using Dolt just requires two steps:

1) add "DOLT" after the call to LT_INIT, AC_PATH_LIBTOOL, or
   AM_PATH_LIBTOOL in your configure.ac or configure.in script, and
2) append dolt.m4 to your project's acinclude.m4.

For any system Dolt does not support, it will transparently fall back
to libtool.

dolt.m4 takes up less than 4kB; it writes out a minimal doltcompile
script which never forks except to run the compiler or to mkdir the
.libs directory if it doesn't already exist.  I have tested it with
various projects, and benchmarked[2] its performance against the same
projects using only libtool.  Results:

kdelibs without dolt: 8m6.115s
kdelibs with dolt:    3m50.065s

gtk+-2.0 without dolt: 2m31.825s
gtk+-2.0 with dolt:    1m33.858s

libx11 without dolt: 1m50.163s
libx11 with dolt:    0m53.417s

libxml2 without dolt: 0m25.722s
libxml2 with dolt:    0m19.576s

dbus without dolt: 0m20.062s
dbus with dolt:    0m8.940s

I have attached a snapshot of dolt.m4 for convenience.  You can also
obtain the current version of Dolt from Git with:

    git clone git://svcs.cs.pdx.edu/git/dolt

or download a snapshot tarball from
<http://svcs.cs.pdx.edu/gitweb?p=dolt.git;a=snapshot;h=master;sf=tgz>.

Please try Dolt with your project, and see if you get comparable
performance improvements.  If you want to make Dolt replace libtool on
your system, feel free to send me a patch to dolt.m4; just remember
the basic tenet that any logic must run at configure time, not build
time.  You can figure out what compiler flags libtool uses by running
"touch dummy.c && libtool --mode=compile gcc -c dummy.c -o dummy.lo";
that will print two compiler command lines, one for the shared object
and one for the static object.

Future directions:
* Support GNU/Linux on architectures other than x86 and x86-64.  I
  think most will work with exactly the same compiler flags, but I
  didn't want to add any architecture I couldn't test.
* Support other systems.
* Possibly try to run libtool on a dummy source file at configure time
  to figure out the necessary flags to use when building library
  objects, but that seems error-prone.
* Replace libtool --mode=link.
* Replace libtool --mode=install.
* Optionally stop installing .la files.
* Make dolt.m4's output of doltcompile cleaner.

- Josh Triplett

[1] "doltcompile" stands for "do ltcompile"; the alternate reading
"dolt compile" led to the name "dolt".

[2] General testing methodology:
* Run ./configure && make && make clean, to make sure it builds and to get
  everything cached.
* Get the "before" time: time make >/dev/null 2>&1
* Remove and re-extract the source.
* Add dolt.m4 to acinclude.m4 and add DOLT to configure.in or configure.ac.
* autoreconf -v -f -i && ./configure && make && make clean, to
  make sure it still builds and to get everything cached again.
* Get the "after" time: time make >/dev/null 2>&1

dnl dolt, a replacement for libtool
dnl Copyright © 2007-2008 Josh Triplett <josh freedesktop org>
dnl Copying and distribution of this file, with or without modification,
dnl are permitted in any medium without royalty provided the copyright
dnl notice and this notice are preserved.
dnl
dnl To use dolt, invoke the DOLT macro immediately after the libtool macros.
dnl Optionally, copy this file into acinclude.m4, to avoid the need to have it
dnl installed when running autoconf on your project.

AC_DEFUN([DOLT], [
AC_REQUIRE([AC_CANONICAL_HOST])
# dolt, a replacement for libtool
# Josh Triplett <josh freedesktop org>
AC_PATH_PROG(DOLT_BASH, bash)
AC_MSG_CHECKING([if libtool sucks])
AC_MSG_RESULT([yup, it does])
AC_MSG_CHECKING([if dolt supports this host])
dolt_supported=yes
if test x$DOLT_BASH = x; then
    dolt_supported=no
fi
if test x$GCC != xyes; then
    dolt_supported=no
fi
case $host in
i?86-*-linux*|x86_64-*-linux*) ;;
*) dolt_supported=no ;;
esac
if test x$dolt_supported = xno ; then
    AC_MSG_RESULT([no, falling back to libtool])
else
    AC_MSG_RESULT([yes, replacing libtool])

dnl Start writing out doltcompile.
    cat <<__DOLTCOMPILE__EOF__ >doltcompile
#!$DOLT_BASH
__DOLTCOMPILE__EOF__
    cat <<'__DOLTCOMPILE__EOF__' >>doltcompile
args=("$[]@")
for ((arg=0; arg<${#args@<:@@@:>@}; arg++)) ; do
    if test x"${args@<:@$arg@:>@}" = x-o ; then
        objarg=$((arg+1))
        break
    fi
done
if test x$objarg = x ; then
    echo 'Error: no -o on compiler command line' 1>&2
    exit 1
fi
lo="${args@<:@$objarg@:>@}"
obj="${lo%.lo}"
if test x"$lo" = x"$obj" ; then
    echo "Error: libtool object file name \"$lo\" does not end in .lo" 1>&2
    exit 1
fi
objbase="${obj##*/}"
__DOLTCOMPILE__EOF__

dnl Write out shared compilation code.
    if test x$enable_shared = xyes; then
        cat <<'__DOLTCOMPILE__EOF__' >>doltcompile
libobjdir="${obj%$objbase}.libs"
if test ! -d "$libobjdir" ; then
    mkdir "$libobjdir"
    mkdir_ret=$?
    if test "$mkdir_ret" -ne 0 && test ! -d "$libobjdir" ; then
        exit $mkdir_ret
    fi
fi
pic_object="$libobjdir/$objbase.o"
args@<:@$objarg@:>@="$pic_object"
"${args@<:@@@:>@}" -fPIC -DPIC
__DOLTCOMPILE__EOF__
    fi

dnl Write out static compilation code.
dnl Avoid duplicate compiler output if also building shared objects.
    if test x$enable_static = xyes; then
        cat <<'__DOLTCOMPILE__EOF__' >>doltcompile
non_pic_object="$obj.o"
args@<:@$objarg@:>@="$non_pic_object"
__DOLTCOMPILE__EOF__
        if test x$enable_shared = xyes; then
            cat <<'__DOLTCOMPILE__EOF__' >>doltcompile
"${args@<:@@@:>@}" >/dev/null 2>&1
__DOLTCOMPILE__EOF__
        else
            cat <<'__DOLTCOMPILE__EOF__' >>doltcompile
"${args@<:@@@:>@}"
__DOLTCOMPILE__EOF__
        fi
    fi

dnl Write out the code to write the .lo file.
dnl The second line of the .lo file must match "^# Generated by .*libtool"
    cat <<'__DOLTCOMPILE__EOF__' >>doltcompile
{
echo "# $lo - a libtool object file"
echo "# Generated by doltcompile, not libtool"
__DOLTCOMPILE__EOF__

    if test x$enable_shared = xyes; then
        cat <<'__DOLTCOMPILE__EOF__' >>doltcompile
echo "pic_object='$pic_object'"
__DOLTCOMPILE__EOF__
    else
        cat <<'__DOLTCOMPILE__EOF__' >>doltcompile
echo pic_object=none
__DOLTCOMPILE__EOF__
    fi

    if test x$enable_static = xyes; then
        cat <<'__DOLTCOMPILE__EOF__' >>doltcompile
echo "non_pic_object='$non_pic_object'"
__DOLTCOMPILE__EOF__
    else
        cat <<'__DOLTCOMPILE__EOF__' >>doltcompile
echo non_pic_object=none
__DOLTCOMPILE__EOF__
    fi

    cat <<'__DOLTCOMPILE__EOF__' >>doltcompile
} > "$lo"
__DOLTCOMPILE__EOF__

dnl Done writing out doltcompile; substitute it for libtool compilation.
    chmod +x doltcompile
    LTCOMPILE='$(top_builddir)/doltcompile $(COMPILE)'
    AC_SUBST(LTCOMPILE)
    LTCXXCOMPILE='$(top_builddir)/doltcompile $(CXXCOMPILE)'
    AC_SUBST(LTCXXCOMPILE)
fi
# end dolt
])