Re: Glad to see interest in install/packaging

[David Elliott <dfe infinite-internet net>]

Morey Hubin wrote:

> On Sun, 29 Apr 2001, David Elliott wrote:
>
>   Just to clarify, I was thinking ahead.  To setup an automated build on
> a build farm you have to know a little about the product you're building.
> Setting the ground rules about what you can guarantee on the user's
> systems reduces the complexity of the automated build environment.
>

To be honest, you seem to be on a totally different level.  That is a good thing.

>
>   I'm concerned with source tree setup, package creation and package
> management.  Less about PC based hardware.  My only contribution far
> as build hardware is that compilations are RAM and disk/IO intensive
> operations.  A dual or quad cpu machine with moderate cpu speed and a
> fast SCSI raid.  I don't know any PC brand names to recommend.
>

Too bad there are no MP Athlons at this point.  I wonder if we could get some
company with an interest in a better Gnome to donate the use of a dedicated
server for this.  I would recommend using Sourceforge but the proposed level of
use would be like a DoS attack.

>
> > Morey Hubin wrote:
> >
> > >
> > >       ) A standard install path that everyone can use universally.
> > >         I setup machines so /usr/gnome is a symlink to the actuall
> > >         Gnome install.  That way I can burn in library paths using
> > >         the linkers -R/usr/gnome/lib (or equivalent for your OS)
> > >         and not have to worry about setting LD_LIBRARY_PATH at runtime.
> > >
> >
> > On Linux at least we ought to follow the filesystem standard. Stable
> > installed packages should have prefix=/usr, all configuration goes in /etc
> > or for lots of files in a directory under etc.
> >
> > However, development releases shouldn't screw with a stable gnome config, so
> > the best place for those is probably in /opt/something.  Perhaps something
> > like /opt/gnome-20010429 for CVS snapshot builds.  The key is to not lock in
> > any one prefix.  I believe KDE still defaults to /opt/kde, but the RH7.1
> > packages of it install directly into /usr (where a stable release should
> > really be on a RedHat system).
>
>   That's fine.  Linux distributions glob everything into /usr/bin.  I can't
> say I'm a fan of that but it works.  If you overwrite something critical to
> your system then get out the Red Hat CD and begin re-installing right now.
> Me, I just /bin/rm -rf /usr/gnome-build2 and all errant bins are forgiven.
>

I have noticed the general trend that most seasoned UNIX users tend to use /opt.
Many Linux users err the other way and put everything in /usr.  To paraphrase
Jeremy White (CodeWeavers) "I prefer to have everything in nice rm -rfable
spaces." (check wine-devel winehq com a few months ago).

The whole point of using RPM is to allow you to glob everything into /usr without
screwing up your system.  The #1 guideline for building rpm packages is to never
ever ever ever ever EVER /EVER/ do it as root (could I possibly have stressed
that point more).  In fact, I think redhat should just ditch the /usr/src/redhat
directory completely.

By using the Buildroot you can have the install phase install into
/var/tmp/packagename/usr and then it uses cpio in the /var/tmp/packagename
directory to put it all into an archive.  RPM will not install a package if it
contains the same file as another package.  So assuming your /usr/bin/tar and
/usr/bin/sed are belong to an RPM package then you have to --force it to make it
install, and that would be a pretty damn stupid move.

>
>   Using /usr/bin is extremely dangerous on networked Solaris boxes.  Solaris
> provides thier own versions of tar, sed, libintl.a and other GNU replacements
> that are incompatible with the originals.  Overwriting one of these can be
> fatal.  In addition Sol uses slices for /usr and /opt leaving the bulk of the
> disk space under /home/<nodename>.  Pre-Gnome configured systems don't have
> the space in /opt (or /usr) for 250Mb+ of new binaries.
>

Of course on Solaris none of that RPM stuff applies and I would DEFINITELY not
use /usr/bin.  Incidently, isn't Sun planning on including Gnome with newer
releases of Solaris?  In that case I would see gnome being under /usr/bin or
maybe go totally nuts like X Windows and put it in /usr/gnome/bin (i.e. like
/usr/X11R6/bin)

>
>   I'm partial to the /opt/gnome-MMDDYY for snapshot builds.  Again we should
> look into choosing a static path to serve as a reference point for the gnome
> run-time.  Say /opt/gnome-devel -> /opt/gnome-20013322 so that we can burn in
> libpaths to /opt/gnome-devel and still allow devel to change quickly.  This
> has a large effect on the build environment so that is why I harp on it.
> I can explain this in detail some other time if anyone cares to read it.
>

I am more partial to the YYYYMMDD format, it sorts better when doing an ls (was
that a typo on your part?).

The more I think about it, the more I like using rpath for snapshots.  However
for release packages I would discourage it.

In keeping with the suggestion below that we not build all packages with the same
frequency it may be more prudent to ditch the date thing altogether and just use
gnome-devel.  Really and truely why in the hell would you want more than one
complete installed snapshot of gnome.  More than likely you would want to upgrade
or downgrade only one or a handful of packages.

>
>   Also keep in mind that ~/.gnome, ~/.gnome_private and ~/.enlightenment
> keep absolute paths to themes, libs and other apps.  Changing from
> /opt/gnome-1111 to /opt/gnome-2222 means that most of your user config
> is gone and enlightenment stop working.  The enlightenment binary actually
> remembers the path where it was installed.  If you move it then it ceases
> to work.  It is not the only exec that does this but it is the most obvious.
>

Good call.  Along these same lines, should we consider using different ~/
directories for snapshot builds?  This will really screw people who want to
switch between a working stable gnome in /usr and the snapshots in
/opt/gnome-devel.  I am not so sure if I like this idea but I am throwing it out,
maybe like ~/.gnome-devel, ~/.gnome_private-devel and so on.

> >
> > >
> > >       ) To prevent a chicken and egg argument I would like to create
> > >         a Perl based Gnome install manager (GIM) that in no way relies
> > >         on Gnome components to run.  I see a PerlTK GUI and an alternate
> > >         silent install control language (really simple syntax) that
> > >         manages the installation of kits.  It may not be as beautiful
> > >         as other Gnome apps but it will be portable and easy to edit.
> > >
> >
> > I don't see adding a dependency on TK as a good thing.  Why not use the
> > latest stable GTK?  It's not really a chicken and egg problem.  The latest
> > stable GTK is pretty much standard issue on Linux now and building GLIB and
> > GTK is a fairly trivial task.
> >
>   PerlTK is a simple downloadable perl module.  Building GLIB and GTK is not
> something non-Linux beta testers can do easliy.  Documentation people either.
> For you and I it's a piece of cake.  The rest of the world is not so savy.
> Most of the people I work with have never heard of configure/autoconf, do not
> have root access to thier machines and do not know about compiling packages.
>

I see your point here.  TK is indeed a very nice lightweight toolkit found on
just about every UNIX system, and a UNIX system without Perl just isn't a UNIX
system :-).  I say that of course with almost no real experience writing Perl
code, although I have certainly read plenty of it.

However, if you are targetting people without root access to their machines then
why have a user-friendly installer at all.


>
>   Anyway when I heard about beta testers I thought about having a simple
> Perl download manager.  Linux is the only OS that delivers glib&gtk natively.
> My thought was that we could piece together a simple download tool that
> stores the names of mirrors.  This whole thing is off topic.
>

Yeah, I have been getting that feeling lately too, so I suppose I'll drop it at
that.

>
> >
> > >
> > >         The key is to identify the common needs for most native install
> > >         managers ( rpm's, tgz's, Solaris pkg's, dpkg's, ...) and devise
> > >         a GIM Virtual Interface (VI).  Aka, a fancy name for a set of
> > >         functions that the GIM needs in your Perl module.
> > >
> > >         Ex:  To install RPM's you write a perl module that knows how
> > >              handle an RPM and also contains the functions we define.
> > >              From then on the GIM will be able to manage the install of
> > >              any RPM.
> > >
> > >              If you want to install Solaris packages then you write a
> > >              module and the GIM does the rest.
> > >
> >
> > This does not really sound like packaging components, but rather
> > distributing them.  I don't think this project is trying to compete with
> > red-carpet or apt-get.
> >
>  +++++++++ Off topic so skip to next question if not interested +++++++++++
>   No more than Slackware's rough ncurses install competes with Red Hat's.
> Your right though, it is off topic.  My companies "beta testers" are generally
> well organized and thurough people who are fairly helpless at a UNIX prompt.
> A simple download&install manager may increase your beta tester population.
>

Now that is not really all that off-topic.  I think it is imperative that gnome
be beta tested on several flavors of UNIX.  Whatever can be done to help this is
most certainly a good thing.  Given that we want people to be able to update to
newer binaries as bugs are fixed this is a very valid idea.

With that said I have a feeling that most debian users are going to want to be
able to simply apt-get this.  I believe that releasing via apt-get is actually
pretty trivial.  For Red Hat users we could either hack red-carpet so we can add
our own channel ( I am assuming this would also be fairly trivial) or we could
simply build a nice stable apt-get for Red Hat which I have actually been wanting
to do for a while but have not because even if I had it, where the hell would
I apt-get from!  I would vote more for apt-get for RPM rather than red-carpet,
less dependencies.

That still leaves other UNIX users in the dark though, including Slackware.  So
at that point you have to make some sort of system that is lightweight and easy
to use.  For this I would agree that perlTK is definitely the way to go.

Okay, so I probably went off topic again... whoops.

>
> >
> > >
> > >         With a little extra effort we could probably add to the VI
> > >         finctions that allow us to work both ways and make new packages
> > >         automatically.  It's really straight forward.
> > >
> >
> > This does not sound trivial at all.  It sounds like you want to rewrite
> > apt-get and make it cross platform.
> >
>  +++++++++ Off topic so skip to next question if not interested +++++++++++
>   I'm well versed in perl modules and would probably take about 2 weeks
> to code and test.  I'm not boasting, they are that easy to write.  I've
> coded enough installations to know most of the basic needs.  If you know
> the list of files and the package dependencies then creating the tgz, rpm,
> dpkg or Sol pkg is routine.  I'm not an expert on rpm's but I'm learning
> fast.
>

RPM is really really cool once you get past the lack of good docs.  Hopefully as
we start to design these packages all of the developers will come out with a
better understanding of RPM packaging.  And hopefully there will be enough debian
users here that we can all come out with a better understading of dpkg.  Although
so far I have not seen any debian users on this list (though it is extremely
new).

>
> > >
> > >       ) A standard path to source that developers can use with GDB.
> > >         GDB expects to find source in the same place it was built.
> > >         If it doesn't then is asks the user to supply the correct path.
> > >         This can get annoying for large projects.  If all debuggable
> > >         builds take place under /usr/gnome_source then developers can
> > >         setup a /usr/gnome_source symlink to thier download area and
> > >         GDB away.
> > >
> >
> > Now this is a very valid point.  I would suggest $prefix/src/package-name.
> > That is something like /usr/src/gnome-core-1.4.0 or for devel stuff
> > /opt/gnome-20010429/src/gnome-core-1.5.0  (assuming 1.5.0 is the devel
> > version and the prefix is /opt/gnome-20010429).
> >
>   To expand a little I have /usr/gnome_source as the root of the source
> tree followed by, [ stable/ || unstable/ || gnu/ || contrib/ || scripts//]
> then the packages below that.  Contrib/ contains jade, Berkeley DB, libjpeg
> Perl5.6 (for glade) and other non-Gnome buildables.
>

Interesting.  I do think it should be a subdir of the install path rather than
over in /usr though.

>
> > >
> > >         This does mean that any build machines will have to mount build
> > >         space under /usr/gnome_source.
> > >         The debugger knows the absolute path to the source so there's
> > >         no way to use symlinks to fake the path to the debug build.
> > >
> >
> > Hmm, that sounds like a real fuck to me.  Is there any sane way around this?
> >
>   Not that bad really.  In my work environment compute servers are special
> and we can do non-standard things to them.  The ends justify the means :)
> As long as the configuration is reproducable and well documented we can
> make changes to improve the environment for developers.
>

I am assuming that if the end user doesn't have enough space on that filesystem
that symlinking /usr/gnome_source to some other directory will work.  Am I right
here?

>
> > >
> > >       ) We sort the list of packages into classes like:
> > >           Core Libs   (stuff heavliy depended upon by user apps)
> > >           Gnome Libs  (stuff needed to run the Gnome App Environment)
> > >           Extra Libs  (stuff no so heavily depended on)
> > >
> > >           Gnome Run   (panel, menu applets ... )
> > >           User Apps   (Useable even if you are not running gnome)
> > >           ... User Apps can be subdivided by what they do ...
> > >
> >
> > I believe that for the most part this is already done.  Really the more
> > fine-grained control the better.  But then again, don't overdo it.
>
>   You can probably optimize build server time by rebuilding core libraries
> less frequently than apps.  Thats why I wanted to identify them.  The more
> dependencies, the less often it should be rebuilt.  Changes to low level
> libs tend to be more invasive, take longer to perfect and are generally
> more dangerous inbetween stable builds.

Yeah, back in the day of Gnome 0.x the core libs changed daily so I have been
thinking more along the lines of keep everything snapshotted from exactly the
same time.  But really we only want to snapshot the cores when there are major
changes.

>
> >
> > >
> > >       ) Release and Debuggable versions of packages wherever possible.
> > >
> >
> > Release early, release often.  Preferably have a high-power compiling
> > machine or farm that automatically builds and packages CVS every night.
> >
>   We build 23 versions of a 4+ million line software package 6 days a week.
> I am a firm believer in automatic turnover and publishing as much as possible.

What the hell are you building?  Emacs? ;-)

>
> > >
> > >     Thats plenty for now...
> > >
> > >   Morey Hubin
> >
> > -Dave
> >
> >
>
>   G'night all
>   Morey

Good idea.  This has been a very cool brainstorming session but it's time to
sleep on it.  (Though I still have a couple other mails to write)

Good night!
-Dave