I wouldn't (publicly) get riled at him, because it looks too much like you've got something to hide. "Don't drop in on me without an appointment, because I need time to hide the still out back..."
However, it IS a waste of his time and a lost opportunity for both of you, for this to happen. I would tackle him over it, without the animosity. Don't pretend you didn't see the card - they would be used to that, it would wear thin. We've had problems with difficult child 3's schoolwork not arriving in the mail and simply telling the school that our delivery is unreliable, is met with sceptical glances. When I produced a letter of apology from the postal service, admitting to the problem, THEN I was believed.
What I suggest you do is make it clear - you've got a short fuse on this, because you've had so many past POs seem to start off well with you, and who then seem to be trying to find 'dirt' on you, rather than help your boy. And when he turned up on spec, it pushed all the wrong buttons and angered you because of the wasted opportunity. You would have been home if you had known; heck, if he had even rung your mobile, you might even have been able to say, "I'm on my way home anyway; stay put, I'll be there in five minutes and put the kettle on."
You and the PO should be working as a team. OK, in the past they did not. You need to tell this guy about that and let him know why you are so prickly and gun-shy about this sort of thing. You have been badly disrespected in the past when all you want is to help your son. Which hasn't happened, despite all your efforts. "So please, Mr PO, can this be different? DON'T drop in on spec, unless you also ring to let me know or at least find out if I'm in the area anyway and can meet up with you. Heck, I'm worth a call on my mobile, because wherever you happen to be, I might be nearby and about to go on a work break, we could meet then. Or we might be able to talk, even briefly. We are a team; let's work as one, work together, and finally, HELP this boy!"
Keep the focus on getting help for your son. Avoid the "Don't invade my turf without my knowledge or permission" because it smacks too much of a hillbilly on the veranda brandishing a huge shotgun, warning off the travelling salesman with "Git orf mah land or I'll blow yore brains out." Yes, we get scenes like that in Australia too...
You have a right to feel prickly about this. But I think you need to shove those feelings aside (even while you let them continue to warn you of the need for caution) and see what this guy has to offer for your son.
Fingers crossed that this time, it will be better.
Marg