Well, technically, I think the buyer is correct, in as much as he's done his bit, presumably satisfactorily once he's paid you. The only thing left to do is for him to rate how well you perform delivering the item. It it all goes pear-shaped, it can't really be the buyers fault unless he lies about the delivery (which is where recorded delivery come is)! I generally would leave feedback as soon as payment is received.
The fact that it bothers this guy, and that he's arguing the toss makes me wonder what his motives are though - you wouldn't think it was that big a deal

In the old days on ebay, sellers would sometimes hold back on giving feedback until the buyer had left feedback for them , which is basically a form of blackmail IMO. I.e. "if you leave negative feedback for me, I'll leave negative feedback for you". Neil will know exactly who I'm thinking of when I mention one particular ebay seller who was famous for this. Ebay termed this "retaliatory feedback" and eventually changed the system because it was becoming a fairly big issue.
I suppose the real point though is that feedback is optional isn't it? You're not
required to leave any at all, just encouraged to, so surely you can do what you like?