This must be so hard on you. Hang in there, he needs some time to adjust and get used to this situation. Of course he is going to want to be with you, you are his Mom! It hurts to hear... you are doing the right thing for now. Give it some time and don't be to hard on yourself.
How are you feeling??? I hope you are taking some time to yourself, I know it is hard. Even a few minutes sometimes helps...