Hi Cosimo,
In my experience, unless we find native infrastructure that someone is willing to lend, there is absolutely 0 gain from running the compilers natively. However, fully cross compiling can be a nightmare (I know this by experience on the Win32 build stuff I did years ago). Fixing the autotooling on every single project is a no-go, and there are build-time tools that are compiled and run locally which makes things really hard for a fully cross compiled environment.
My suggestion would be that you run the ./configure && make stage on an ARM environment that (virtualized or not, the important thing here is that it has decent I/O performance, i.e: not a RaspberryPI 2) and that you use a distcc-like solution on your existing hardware and a cross compiler at the builder's end.
My 2 cents.