I don't think it will damage the screwdriver, but the unfortunate thing is that you really should use the right bit for the right screw, or you run the risk of stripping either or both.
Removable bit driver gizmos are easily obtainable online, and honestly would be a worthwhile investment if you deal with that many different screws.
I would say that using a phillips on a pozidrive screw is not the worst thing you can do, but remember, with that much more metal carved out of the head of the screw, and no metal from the screwdriver to re enforce it, stripping the screw head a very likely possibility.
Def