Re: [Vala] Chunking a file in multiple chunks

[Guillaume Viguier <guillaume viguierjust com>]

Hi,

Thanks for all of this. It solved my problem.

Guilluame Viguier-Just

El sáb, 06-11-2010 a las 08:33 +0100, Jan Hudec escribió:
> On Thu, Nov 04, 2010 at 20:24:21 -0500, Guillaume Viguier wrote:
> > Hi,
> >
> > I'm totally new to the Vala language, so there might be something
> > obvious I'm missing. I'm trying to chunk a file into different chunks of
> > 500 k (in order to send it over the network) and then calculate its
> > md5sum (to send the md5sum over the network to a service to allow it to
> > verify the file was sent correctly).
>
> Strings in Vala are ANSI C NUL-terminated strings. Their length is not kept
> around in any way, but is determined by looking for the first 0 byte. You
> are, however, reading binary data that can legally contain 0 bytes. Therefore
> you have to use 'uint8[]' type (or the ByteArray type), never 'string' type.
> For arrays, vala keeps extra variable with the length.
>
> > Here is the code I've written so far:
> >
> > var file = File.new_for_path(source_file);
> > var file_stream = file.read();
> > var data_stream = new DataInputStream(file_stream);
> > data_stream.set_byte_order (DataStreamByteOrder.LITTLE_ENDIAN);
>
> Why are you creating a data_stream here? The FileInputStream you got from
> file.read() is already an InputStream and that's all you seem to be using
> anyway.
>
> > size_t chunk_size = 500000;
> > size_t bytes_read = 500000;
> > string[] chunks = new string[0];
> > uint8[] chunk = new uint8[chunk_size];
> > string photo_data = "";
> > var bytes = new ByteArray();
> > string original_sum;
> > string chunk_string = "";
> > do {
> >         data_stream.read_all(chunk, chunk_size, out bytes_read);
> >         chunk_string = (string) chunk;
>
> This is wildly incorrect. chunk.lenght is chunk_size, but chunk_string.length
> is just the length to the first NUL byte. What you want is bytes_read long
> slice of the array, as an array.
>
> >         bytes.append(chunk);
>
> This is almost correct. It is not, because bytes_read may still be less than
> chunk_size (at end of file). So you need to append a slice (for which I don't
> remember the syntax).
>
> >         chunks += chunk_string;
> >         
> >         debug("bytes read: %Zd", bytes_read);
> >         debug("string size: %Zd", chunk_string.size());
> > }
> > while (bytes_read == 500000);
>
> I'd compare it to chunk_size when you already have it. Than your code will
> still work if you change the size above.
>
> > original_sum = Checksum.compute_for_data(ChecksumType.MD5, bytes.data);
> > debug("Original sum: " + original_sum);
> >
> > The number of bytes read in the first debug call seems fine (the file
> > I'm trying to read is 1.9 Mb and I get 4 "bytes read" outputs, which
> > correspond to the size of the file). However, the "string size" seems
> > wrong (the first chunk gets a size of 4, the second one 25 etc... and
> > finally, the original_sum computed is totally wrong as well (does not
> > correspond to the md5sum of the file).
>
> Well, where did you tell the string the value of bytes_read that you expect
> it to be of correct size?