I don't know if this will help - the "what I would have done" routine. Chances are you already did this and still had problems.
WHat I would have done when I found the toys not put away as I expected - I would have said, "I asked you to clean up your toys, you just moved the mess from here to there. You know what I mean, let's try this again. Get this done right so we can go for that swim."
I think the problem was, he was so eager for tat swim he didn't want to delay it. He not only shifted the toys, he got his swimsuit on. He was eager. Too eager, I think. Which meant that the frustration when he got called on not doing it right, is what boiled over. You need to go a little bit easy on this kind of frustration, but no way can he get away with damaging the place. For us, the consequences of doing this sort of damage, requires restoration.
Example - difficult child 3 got angry at his grandma's and went out, slamming the door hard. Glass in a small window beside the door shattered. difficult child 3 hadn't hit the window directly, but the force of his slam was responsible so he had to help fix the damage. He had to work with husband to measure the window, had to go find a piece of glass, had to pay for the glass (if it wasn't too expensive). End result was, grandma decided that little window was too risky to replace and asked for a board there instead. So difficult child 3 was made to do it under husband's supervision. He had well and truly calmed down because this took a week to get organised, but he still had to follow through because it was his temper that had caused the damage.
Again the other night difficult child 3 was cranky because he had dropped the stylus from his Nintendo 3DS in his very messy bedroom and couldn't find it. When he left to come down to Grandma's (we go to dinner there every night) difficult child 3 slammed the gate really hard and the latch broke. Next day husband got difficult child 3 to work with him to replace the latch and repair the gate. As husband said, nobody had been hurt by the outburst but nobody should be inconvenienced either, when they hadn't been the ones to do the damage.
Along the way, difficult child 3 not only learns that impulsive actions have consequences, but also the skills to make repairs.
As I said - I'm probably not telling you anything you're not already doing yourself. If you are, then at least my story can help you realise you're not the only one going through this and spectrum kids especially can really have problems with transitioning. In your son's mind, he was already in the car on the way to go swimming. Everything else was a temporary inconvenience and he really could not understand, AT THAT MOMENT, why you "broke your word". But when he's calmer, you explain. However, in his mind he may still feel that he had no choice but to use the hammer because if you had not refused to leave immediately the toys were gone from where he had left them, he would not have gotten so mad. That is how Autism Spectrum Disorders (ASD) kids often think and trying to break that thought pattern cannot be done by willpower and obstinacy.
There are steps, levels of arousal that we need to be aware of and learn to de-fuse. When he shoved stuff out of sight and thought that was enough, it indicates that the task required was eclipsed by the proposed reward. Which perhaps should not have been used as a reward since it was so highly valued. When the reward is so much bigger than the task, sometimes the task gets skipped. It can seem illogical to us, but kids can get over-excited and lose track.
But 20:20 hindsight is a wonderful thing. We can get judgemental, we can feel guilty, or we can simply pick things up and keep on going. It's what we do when we love our kids.
Sorry about the door - I would make sure he is involved at some stage with repair, even if it's just painting the new one. But not in any sense of punishment, but simply as a job to be done. If he seems to enjoy the work - good! And never forget to thank him or praise him for it, and do not keep reminding of the damage. Believe me, he will not forget. The lesson, the best lesson, is learning to make something good out of something bad, and this includes feelings.
Marg