Re: bmap for raw image deployment

[Artem Bityutskiy <dedekind1 gmail com>]

Hi,

On Thu, 2015-05-21 at 09:17 -0700, Jasper St. Pierre wrote:
> On Thu, May 21, 2015 at 6:36 AM, Artem Bityutskiy <dedekind1 gmail com> wrote:
> > Yes, just to give an example of how this idea was actually used in real
> > life - you are mass-producing your phone, they all have the same eMMC
> > (or few variants), you have a single image which your flasher deploys on
> > the factory. The flasher is very dumb and just writes blocks. Bmap file
> > tell which blocks to write.
> >
> > But with a smarter flasher or the "firstboot" service one could enlarge
> > partitions or volumes if needed (in case the image is smaller than the
> > storage device).
>
> This is actually super useful for us. We have a similar scenario at
> Endless. We use 'dd' to flash initial devices, and then do a resize at
> firstboot.

Right. And your image has a set of blocks which have useful data, and
set of blocks which do not have any data (e.g., empty FS space). dd will
write all blocks. bmaptool only the blocks with data, hence the
speed-up.

Now, bmaptool is an implementation of the idea for Linux. What I
mentioned as the real life usage was a different implementation of the
same for mass phone production on a real factory.

>  Currently, this is an ad-hoc bash script in
> https://github.com/endlessm/eos-meta/blob/master/eos-tech-support/eos-resize-device
> -- does bmap include a more robust firstboot resize system?

Bmaptool is agnostic to what is inside the image. The firstboot stuff is
something inside the image, and may be anything. Bmaptool is about how
you make sure you write your raw image to the storage media very very
quickly.

There are 2 parts:

1. Creation of the bmap file: when the image is generated, we can create
the bmap file which basically lists the blocks which have useful data.
2. Writing the image: we use the bmap file to deploy the image faster.

E.g., it is not untypical to have a 32GB image, with partitions and
different file-systems inside. But out of the 32GB there is only 2GB of
useful data, the rest is the unused space. DD would write 32GB. bmaptool
writes 2GB.

Now, this is the basic idea, explained in more details in the docs.

On top of that, there are "goodies" like:
1. SHA256 of the data, so that you verify integrity while flashing,
optionally
2. Possibility to GPG-encrypt the bmap file, then image+bmap file is
securely protected: signature makes sure bmap file was not altered,
sha256 inside bmap file make sure the data in the image was not altered.
3. And more, see docs.

> We'll certainly look into using this internally, thanks for the pointer!

Sure. I think bmap stuff is an awesome idea certain problem space, and I
saw it it work. I personally do not use it anymore, but I'd be happy to
see it being useful for others.

Thanks!