Warning: Untested
If you build/compile GIMP and key libraries (e.g. GEGL, BABL) from source, maybe you should try
CFLAGS="-march=native -O3" CXXFLAGS="-march=native -O3" ./autogen.sh ...
instead of the usual
./autogen.sh ...
This may, or may not, make a noticeable difference.
But if I was concerned about speed on my specific hardware, I'd try that first.
And actually, if I do this with BABL and GEGL, make check within GEGL runs 26% faster on my modest laptop.