If you've heated up the steel sufficiently to discolour it, I think you may well have changed it's properties
I suppose one end of the rivet is peened down, while the other head was part of the original rivet. I would guess that the peened down end may be the tighter end if the rivet diameter expanded during peening. So would that suggest tackling the other end to grind down?