The only downside I found when using GNOME with a11y enabled on my system was that somehow the mousekeys were enabled by default. This is very irritating when trying to login at my bank. With a single keypress on your numeric keypad you don't see the mouse moving that much, so you assume there's something flaky with keyboard setup again (yes, I also have a very bad experience with XKB).
The AccessX/XKB stuff is different from AT-SPI, and each is enabled differently. The AccessX/XKB stuff is an X Server extension that gives you SlowKeys, MouseKeys, etc., whereas AT-SPI is a graphical toolkit and desktop infrastructure gives you the ability to have higher level accessibility support for testing and assistive technologies. This proposal is about enabling AT-SPI support and not AccessX/XKB.
Note that this isn't meant to be a "you stupid dope" note. :-) Like I mention, it's come up more than once, so there's obviously some mental conflation going on about accessibility support being one big blob. That's completely understandable, and I hope this note helps clarify that AccessX/XKB and AT-SPI are at least different.
Will