Can you drop off the 3yo for a playdate with a peer for an hour once a week? Even better, can you afford something like a moring preschool a couple days a week for the 3yo? That could buy you some much needed down time.
Check out your local churches for mother's groups, too. Where I live they have an organization called MOPS (Mothers Of Preschoolers) that meets weekly at churches for an hour or so once a week. The little ones get to play together supervised by moms who rotate volunteering, while the moms have time to hear a speaker, do a craft, or have some fellowship time. There is also an organization called MOMS Club that may have a chapter near you. They are geared towards stay at home mom's and usually have informal monthly meetings, weekly park days, playgroups, monthly Mom's Night Out, etc. I found that invaluable for preserving my sanity when my kids were young and undiagnosed yet!
As for the guilt of leaving the house -- fuggeddaboutit! It does not help you, it does not help your family. If you DON'T take care of yourself, you CAN'T take care of your family. If you don't have a therapist, consider finding one you can check in with, if only once a month. They can be worth their weight in gold when you are going through a rough patch -- I know it's helped me because my friends don't always have the bandwidth to talk about the issues I have in my life with difficult child's. And a therapist will. And of course, come here, come often! Because you KNOW we all get it. 
