What a uncomfortable situation. Your little guy expressed his feelings & probably felt unheard. Having said that, you are the parent & have every right to invite whoever you feel is appropriate to this party.
difficult child, on the other hand, is 6. He will probably have a love/hate relationship with this young boy or others throughout his early & middle childhood. I see it all the time in the tweedles (who are emotionally more your PCs age). Heck, it's a rite of passage. I see it as a life skill - learning how to negotiate, play & decide what people you want to spend time with or who you want as friends.
I'd continue to invite any child you care to - it really is a learning experience for our children.
