Re: possible optimization

[Federico Mena Quintero <federico ximian com>]

On Fri, 2006-02-24 at 10:30 +0100, Alexander Larsson wrote:
> I just realised that when calculating the "size" calculation for folders
> (the number of items in the folder) we're currently getting all
> information from the files in that folder (i.e. a stat call per file)
> when we only really need the filenames.

You don't even need the names.  stat("/some/directory") will give you
the number of entries in the directory in the st_nlink field.  An empty
directory has st_nlink=2 because of "." and "..".  This is useful for
non-deep counts.

Radek Doulik also pointed me to the interesting fact that Linux provides
more than dirent.d_name for some filesystems; mail attached.  We could
use dirent.d_type to avoid stat()ing sometimes.

So with st_nlink plus d_type==DT_DIR, we could do deep counts very
quickly on at least ext2/ext3.

  Federico

--- Begin Message ---

• From: Radek Doulík <rodo novell com>
• To: Federico Mena Quintero <federico ximian com>
• Cc: kris babi-pangang org
• Subject: Re: d_type in dirent structure
• Date: Thu, 19 Jan 2006 16:44:19 +0100

On Wed, 2006-01-18 at 09:29 -0600, Federico Mena Quintero wrote:
> On Wed, 2006-01-18 at 15:14 +0100, Radek Doulík wrote:
> 
> > I run into that yesterday, when trying to speedup OOo. I didn't know
> > that Linux's dirent structure (came from *bsd) can contain d_type field
> > with entry type similar to/part of stat's mode field. (it doesn't work
> > on all filesystems though - ext[23] works, reiserfs/xfs/nfs doesn't)
> > 
> > In OOo I can now in some cases avoid stat calls. I wonder if your
> > filechooser uses/might use that as well?
> 
> Oooh, very interesting!  I didn't know those fields existed.
> 
> I guess testing for the presence of d_type can be solved with some
> configure.in magic.

In OOo I test for _DIRENT_HAS_D_TYPE inside my code with ifdefs
like
#ifdef _DIRENT_HAVE_D_TYPE
...
#else
...
#endif

The _DIRENT_HAVE_D_TYPE is defined /usr/include/bits/dirent.h, which is
included from dirent.h, so it is enough to include dirent.h.

>   Do you know if there is a way to test which file
> systems support it?

If the filesystem doesn't provide the info, the d_type is set to
DT_UNKNOWN in which case one must fallback to stat again.

So far I know that ext2, ext3 and nfs supports it, while reiserfs does
not. (Which is unfortunate, given that SUSE defaults to reiser - might
be good argument for switching to ext3 though)

> Thanks for the tip; I'll definitely play with this at some point.
> 
> [Kris, sorry for not having reviewed your async branch yet --- I'm in
> Boston this week, and I think I may be able to get to it next week.]
> 
>   Federico
> 
-- 
Radek Doulík <rodo novell com>
Novell, Inc.

--- End Message ---