Re: [Gimp-developer] deriving transform by comparing image before/after edit

[Adalbert Hanßen <Adalbert Hanssen gmx de>]

Bill,

unfortunately the result of a convolution operation can not always be
inverted. The most simple example is a convolution matrix (=folding
matrix) with all zeroes. It definitely wipes the whole image and that
can not be undone!

Unfortunately, the situation is such that the result of a matrix
operation almost never can be inverted. Here an explanation with a
little bit more reference to mathematics, leaving out the details of not
square sized pictures and convolution matrices like those in Gimp, which
only have up to 5*5 entries (think of them as being larger, expanded
with all zeroes around it to all sides to pad them to the image size):

Let M denote the Matrix and FT(M) the fourier transform of it (expanded
to the image size). The Fourier transform is a terribly complicated
thing and in fact, the result of a real valued picture becomes a complex
valued picture, i.e. two pictures: one for the real part and another one
for the imaginary one. Fortunately in mathematics (but not in Gimp)
there also is an inverse Fourier transform which is as complicated as
the FT itself, but once one has a program for FT, one also has one for
the inverse FT. Of course, the FT and its inverse operate on complex
valued "pictures", but you normally only look onto the real part of them
in the non-transformed image. (The FT image is often referred to as in
the frequency range: small and "high frequency" details in the picture
become large in the FT and vice versa, it is all really "convolved", as
one may say in the double meanig of this word).

Let P denote the picture and FT(P) the fourier transform of the picture.

Let M#P denote the convolution of M and P

Then FT(M#P) = FT(M)*FT(P) (point-wise multipliaction)

If FT(M) has entries with zeroes in it, these zeroes will cancel their
respective pixels in FT(P). This of course means that FT(M#P) can no
longer be FT(P). Unfortunatly almost all FT(M) would have zeroes
somewhere, if the convolution matrix would have image size.

The matrix convolution can be computed very effectively and it is quite
powerful, especially for small matrices. Fortunately it now works in
Gimp 2.10.22 (it did not work in earlier versions of 2.10) and
fortunately now we can load and store the convolution matrices.

I frequently convolve scans of documents with something like this

0    0    0    0    0

0   -1   -1   -1   0

0   -1    c   -1   0

0   -1   -1   -1   0

0    0    0    0    0

wherer c ins the centre-weight, a number which is larger than the
negative value of the sum of all other (negative) weights, e.g. c
somewhere in the range 9...12.

This matrix convolution subtracts from each pixel a weighted mean of the
surrounding ones. The purpose of such a filter is to remove background
like shadows or stains from the document before I apply OCR to it
(that's why I do this operation). By choosing c larger than the negative
sum of the surrounding ones, you can give the original picture more
emphasis than to the reduction of the background.

My convolution filter has some "differentiating property" which lets it
emphasize noise in the picture. In order to mitigate that, one can first
apply some gaussian filter. Another way is to integrate the the blurring
(local average) into the convolution matrix, i.e. simultaneously do some
kind of smoothing for the centre, for example

-1  -1  -1  -1  -1

-1    c   c   c  -1

-1    c   c   c  -1

-1    c   c   c  -1

-1  -1  -1  -1  -1

with some appropriate centre weight such that the sum of the c's ie a
bit larger that the negative sum of the -1's around. The centre part is
a box convolution which makes the image unsharp. (It would be nice if
one could change all the centre- weights in one place, but once you
think about that, you trigger the desire to shape the centre weights
like a bell shaped function, which comes closer to gaussian unsharpness.
Indeed, Gaussian unsharpness can almost be approximted by several times
folding with a box function with only positive values).

Of course it would be great, if the convolution matrix could be made
larger than 5*5 for such folding matrices to remove background.

For removing stains and shadows, it would also be beneficial if one
could apply some gray level transformation to each pixel of the image
(e.g. given by a polynominal) before the pixel value is mutiplied with
the matrix entry in the convolution operation and to apply some other
gray  level transformation (one which mostly anihilates the first one)
on the way back before the convolution image is shown.

In addition I would whish me something where I could apply two
convolution matrices to an image: the first one acting like now, but the
second one just to compute some "comparison value" which is used to set
the pixel to "black" if the outcome of the first mentioned convolution
is greater or equal to the outcome of the second convolution. Think of
the second convolution going to some "alpha channel".

But it would become necessary to be able to store and reload all those
matrices and the coefficients of the gray level transformations and the
coefficients of the comparison operation with that second convolution 9n
norder to be usefull.  Great things could be done with Gimp if there
would be some easy to perate playground for such image manipulations!

Try my background removal operation with the appended example file,
which is just a small part from an image taken from a newspaper. Play
with it with different values of the centre weight. Of course, it would
be best to first reduce the immage to a grayvalue-only image, i.e.
without colours.

Adalbert


Am 27.12.20 um 07:52 schrieb Bill Ross via gimp-developer-list:
> Given a pair of before/after jpeg photos edited with global-effect
> commands (vs. operations on selected areas), is it possible to derive
> transformation matrix/matrices that reproduce the result from the
> original? Presumably by iterating over all the pixels in a 1:1 mapping.
>
> My hope is to train neural nets to predict the matrix operation(s)
> required. Example:
>
> http://phobrain.com/pr/home/gallery/pair_vert_manual_9_2845x2.jpg
>
> Thanks,
>
> Bill