Rebelmommy, I understand. You don't need to justify your actions here. It's easy for outsiders -- who have never never lived with your child and don't know the full story -- to judge, but you have to consider your health and safety and the healthy and safety of your other children.
In many ways I feel the same about my son (who is also 24). I love him, but I don't like him a lot of the time. Living with him (which we are at the moment) is just about unbearable. I'm counting down the days (16 to go!) until he moves out.
Hang in there and don't feel guilty. Stick to your boundaries and do what's right for you.