My kids' elementary school had a trip to Boston. I went with oldest boy because he wanted me to. He had a great time and the kids were mostly well-behaved. The PARENTS, however, were awful. My son, who with ADHD inattentive and mild Asperger's usually notices nothing, commented on the mother (an elected official in our town!) who kept going up and down the steps at Fenway Park with multiple beers! If MY kid noticed it... Then there were the dads who had to be rounded up out of the bar near Fenway by the allegedly "connected" dads in the group - who were the best behaved, the most gentlemanly and the politest of the men! The trip was 2 days and one overnight. My D did not go because she was in private school.
We moved difficult child and the 2 younger boys to a different elementary school in town. Their 5th grade trip was to D.C. This school used staff for the trip and did not take parents at all. They called in subs as needed. The principal wanted H or I to travel down separately and stay in a room in the hotel - just in case. difficult child's teacher, a 6'7" angel of a man, said no and that he would be responsible for difficult child. He believed that difficult child would rise to the occasion. difficult child was perfect - one of the best kids on the trip. PC15 went on the same trip two years later and also had a great time.
After the recession hit, the school switched to a day trip into the city. Youngest boy wanted to go on his own so I didn't buy a ticket. They saw Wicked and the wax museum, NBC tour, etc. When I dropped him at the train station to meet his class (his school went with our former school, which dropped the trip to Boston), he decided he wanted me to go after all but it was too late. I told him I would be 15 minutes away at my office and he could call if there was a problem, I didn't hear from him and he had a blast. He did insist that I go as a chaperone the next year when Drama Club went in to see a show. He told the teacher it was my birthday so I got selected for a free ticket (it really was my birthday!).
In your situation, I would go if your son doesn't act up more when you're around (difficult child often did), if he seems genuinely enthusiastic about you going and if you have the time and the money. I would not ask for a group of kids other than him because most parents who go do have their own kids and I think the other kids would notice if your son's own mom didn't seem to want him in her group. I would stay home rather than not have my child in my group.
Hope the trip is fun - Difficult Child is great. I still have to get youngest boy there.