My son, who is very shy and doesn't like crowds watching him, tried out for Little League many years ago. He made it and got on one of the best teams and everyone at school knew it. After a few practices, he came home and said it was too stressful, he was crying while playing, and that he wouldn't continue. We tried what we could to change his mind, but he flat out told us he wouldn't play, even if we drove him there. I can't say it changed his life much that he didn't play. He has always had over the top anxiety and he still does and is on medications at age 34 and playing Little League wouildn't have changed that. But he is also a VERY successful salesman! I know it doesn't make sense, but he knew what he could handle and what he couldn't handle. He does sales mostly in the office, as he still doesn't like to be in front of a group, but when he has to do it, he does. I never understood forcing a kid to do something (an activity) that he didn't want to do. I always said, "Try it and see if you like it." I understand the concept of not quitting and/or not letting your teammates down, however with difficult children I think that those "other circumstances" that drive our kids to make the decisions that they do need to be weighed and that mental health comes first. I know not everyone will agree with me, but, really, I always let my kids give up activities they didn't like, and my grown kids did not grow up to be quitters or slackers and both had been difficult children at one time.
If it were me, I wouldn't sweat it too much. It really depends on his reason for not wanting to continue. by the way, I don't feel like you are an irresponsible parent. There are no playbooks on the "dos" and "don'Tourette's Syndrome" of raising difficult children. Sometimes just getting them to brush their teeth once a day is a victory.