Well- ELectric OXidating aluminium creates a thin yet way more durable layer on the very weak aluminium (Aluminium oxides are among the most durable materials you can use for sanding materials).
If you remove this coating, the handles will wear off just a bit quicker. IMHO it'll be very hard to find a means of eroding the annoying orange (o tempora, o mores) without destroying at least a part of the texture. I've seen a couple of aloxes which had obviously been dremeld with a wire brush - which looked rather messy to me.
Good luck!
Tomcat