Re: Gio & memory management

[nico <nico yojik eu>]

Hi,

Thank you for your patch, Enrico... now the code is smart.
But there is no modification about the memory usage...

John, the "root" of the program is normal... I checked output files (and 
the program end, when directories are not too big, for example my 
home/Project/ directory... I just need about 100 BM to run the program).

I told you, that I've generated graph... I would like to show them now.

Here, are the graphs for the php script:

http://yojik.shtooka.net/profile_php.pdf
We've got 3 charts:
- the first represent the memory usage after each iteration of the 
scan_dir function
- the 2nd represent the "depth" e.g. the number of subdirectories (after 
each iteration of the scan_dir fct)
- the 3th represent the number of scanned files (after ech iteration of 
the scan_dir fct)
- The 4th represent the number of files in memory... (number of files in 
the root dir + number of file in the 1st subdir + in the 2nd etc...)

For the C version we've got these charts
http://yojik.shtooka.net/profile_c.pdf

We can see, that the memory usage only grow up... even when the depth 
decrease...
I stopped the program after ~5000 iterations because my computer started 
to swap (I've got only 512 MB of RAM)

I join to my mail the manually patched C/Glib script. If someone want to 
test is on his own computer (for exemple on the /usr/ directory)...

for information, I give you the php script :

#!/usr/bin/php5
<?

function list_dir($path, $files_in_memory = 0, $depth = 0) {
   global $count;

   $enumerator = scandir($path);  //I want to use a maximum of memory, 
so I don't use the opendir/readdir method
   $files_in_memory += count($enumerator);

   if (is_array($enumerator)) {
       foreach ($enumerator as $filename) {
           $count++;
           if ($filename != "." and $filename != "..") {
               if (!is_link($path.$filename) and is_dir($path.$filename))
                   list_dir($path.$filename."/", $files_in_memory, 
$depth + 1);
               else
                   ;//echo "$filename\n";
           }
       }
   }

   echo (memory_get_usage())."$usage, $depth, $count, $files_in_memory\n";
}


error_reporting(0);

if ($argc == 2)
   list_dir($argv[1]);






John Vetterli wrote:
> Hi!
>
> The problem may be a simple one: the "." and ".." directories.  Unless
> g_file_enumerate_children filters these out for you (the online
> documentation makes no mention of doing so) you need to ignore these.  
> Otherwise, you'll just keep scanning the same directory, or its parent,
> over and over again.
>
> Have a look at the program output.  Do you see "ENTER IN xxx" with the
> same directories repeatedly?
>
> HTH
> JV
>
>
> On Sat, Oct 11, 2008 at 02:06:49PM +0200, nico wrote:
>> Hello,
>>
>> I wrote a program that list recursively files from a given directory.
>> I'm impressed by the memory usage of such a program (after one minutes,
>> it uses more than 100 BM on my /usr/ directory... after 5 minute I have
>> to kill the process : I've only 512 MB of RAM... so my system is
>> becoming unstable).
>> (for comparison, the same program written in php uses less than 1 MB to
>> list all files of my /usr/).
>>
>> Is something wrong in my code? Do I forgot to unref or free something?
>>
>> Regards,
>> Nicolas
>>
>>> #include <glib.h>
>>> #include <gio/gio.h>
>>> #include <stdio.h>
>>> #include <string.h>
>>> #include <stdlib.h>
>>>
>>>
>>> void scan_dir(char const *path) {
>>>     GFile *file;
>>>     file = g_file_new_for_path(path);
>>>     GFileQueryInfoFlags flags;
>>>
>>>     GError *error;
>>>     error = g_error_new(g_quark_from_static_string("TEST_ERROR"), 0,
>>>     "Error");
>>>
>>>     GFileEnumerator *enumerator;
>>>     enumerator = g_file_enumerate_children(file, "*", flags, NULL,
>>>     &error);
>>>     if (enumerator != NULL) {
>>>         GFileInfo *file_info;
>>>         while ((file_info = g_file_enumerator_next_file(enumerator,
>>>         NULL, &error)) != NULL) {
>>>
>>>             const char *filename;
>>>             filename = g_file_info_get_name(file_info);
>>>
>>>             GFileType file_type;
>>>             file_type = g_file_info_get_file_type(file_info);
>>>
>>>             if (file_type == G_FILE_TYPE_DIRECTORY &&
>>>             !g_file_info_get_is_symlink(file_info)) {
>>>                 char *child_path;
>>>                 child_path = (char*) malloc((strlen(path) +
>>>                 strlen(filename) + 1 + 10) * sizeof(char));
>>>                 strcpy(child_path, path);
>>>                 strcat(child_path, filename);
>>>                 strcat(child_path, "/");
>>>
>>>                 printf("ENTER IN %s\n", child_path);
>>>                 scan_dir(child_path);
>>>                 printf("EXIT FROM %s\n", child_path);
>>>
>>>                 free(child_path);
>>>             }
>>>             else {            
>>>                 printf("%s\n", filename);
>>>             }
>>>         }
>>>     
>>>         if (g_file_enumerator_close(enumerator, NULL, NULL) == FALSE)
>>>             fprintf(stderr, "Can not close enumerator\n");
>>>
>>>         g_object_unref(enumerator);
>>>         g_free(file_info);
>>>     }
>>>     g_object_unref(file);
>>>     g_error_free(error);
>>> }
>>>
>>>
>>>
>>> int main(int argc, char** argv) {
>>>     g_type_init();
>>>
>>>     if (argc != 2)
>>>         return 0;
>>>
>>>     scan_dir(argv[1]);
>>>     
>>>     return 0;
>>> }
>>
>> _______________________________________________
>> gtk-app-devel-list mailing list
>> gtk-app-devel-list gnome org
>> http://mail.gnome.org/mailman/listinfo/gtk-app-devel-list