Or go to see him. It will not be on your turf and you can leave when you want to. You and M could stay in a hotel.
You do not need to explain to M or M's sister. Your son can see you and visit. You would also have alone time with M. You could shop, explore, and sight-see. You have a 'built-in' out.
Either way, YOU decide what is best for you!