You feel guilty because you have been fed a load of bovine excrement and so far you have swallowed it. Don't feel bad, lots of us have. It is hard to let our adult kids go and be adults, especially when they show no signs of actually wanting to do so.
Your primary responsibility is to your minor children. Your oldest is a grown man. He is high and wants to come into your home? He can go get a motel room. Your mother is WAAAYY out of line in my opinion. Your oldest has absolutely no business being high around you or his siblings!!! Where is his respect for his family? Why are you supposed to support a full grown adult who has enough money to get high? If he can buy drugs, he can pay for food and shelter for himself.
Every single penny that you give to someone using drugs ends up helping to support their habit. If you house them, then they can spend their money on drugs. If you feed them a sandwich, that is $5 they can spend on drugs. I know that is harsh, but it is also reality.
Your son is sick with addiction. Sadly, it is the only disease that infects the entire family and tells them they are not sick. Please, I beg you, go to AlAnon or NarcAnon Family Meetings. Post here and read, we understand. You are feeling the FOG of Fear, Obligation and Guilt. I know I sound harsh, but I know you cannot just cut him off 100% cold turkey. I don't even mean to go no contact. I mean you need to detach in a loving way and give him as little financial support as possible. You cannot control what your family gives him, but you can control what you give him. Alanon type meetings really can help with understanding all of this.
Reading the article on detachment posted at the beginning of this forum also helps a lot. I can recommend some books that are helpful also. At least I found them helpful.
Let your Mom have her opinions. Know that allowing your son to move back home would do worse than just hurting you. It would seriously harm your younger children. It also would hurt your son. Plus if CPS (Child Protective Services) knew you let someone who was actively using drugs move into your home with your very young children, they would get involved in your life and could take your kids away. CPS is super unpredictable and they don't have to have a provable reason to take your kids. Their workers just have to say they see an imminent danger and then they can take your kids. Usually the CPS workers are fresh out of college with very little training and changing their minds once they decide your home is 'dangerous' is very difficult. In my county it is usually several months before you can get your kids back.
I would use the threat of a CPS investigation to keep your mom off your back if you just cannot handle her nagging. Or else tell her that you would allow someone who is high and disruptive to live with your minor children about the time your head spins around like that girl in the Exorcist movie. If your son not having a place to live is such a big deal to your mother, why not send him to live with her? It seems like she is very concerned, so she should be glad to have him stay with her.
Either she actually will have him come stay, and then she will realize why you don't want him in your home, or she will change the subject. If she tries to change the subject, tell her that until she wants to live with him, she can stop telling you that you have to baby a full grown man.
Why does someone have to provide a home for him anyway? He is 23. He should be working. I can see helping him if he is in school or is disabled, AND he is not behaving badly. Given the getting high and coming in at all hours in a way that disrupts everyone, he wouldn't get help from me. So why isn't he able to support himself?