Jan, if/when I say that sometimes what we do for our kids is wrong, I mean it is wrong for that child. It is not intended as criticism, in fact parents often have been doing their utmost to be consistent, firm disciplinarians. With any other child, it would give a great result. But as you have already found out (and we did too, the same hard way!) the expected "good parenting" methods can seem to make some kids worse. Not your fault.
When I commented that someone has been perhaps treating your son overly harshly, I did not necessarily mean you guys. In our case, it has often been teachers at school whose "putting in their oar" that has undermined a lot of good progress we were making as parents. Also in our case, family members and friends at times try to interfere especially when they think we are not watching them. It's something that drives me nuts, when I leave difficult child 3 in a room with an otherwise intelligent adult family member, who knows (because we have told them) how NOT to handle difficult child 3, and they go do it anyway, because "obviously the parents are just not strict enough". It's as if these idiot adults believe that they will apply discipline A and instantly the child will make miraculous behavioural progress, at which point the friend/family member can say to us, "I know you said not to but I just knew you were wrong so I did it my way and look what a marvellous result I got. Now, here's how to do it for yourself, I will magnanimously help you..."
What generally happens is not so rosy as the well-meaning person expects. They then get resentful when their "I am trying to do something nice as a surprise" backfires badly. And then I am asked to make my child apologise for what often turns out (from his perspective) to have been extreme provocation or teasing. "He's got to learn to take being teased" is what I then get told - but for an Autism Spectrum Disorders (ASD) kid, being teased by an adult is a very bad thing, if they don't understand. We spent years working carefully to avoid mixed messages. When giving directions or answering questions we learned to say, "You're correct," and not "you're right," because "right means "not left" to a difficult child who can only accept a single meaning to a term. An adult trying to make a joke out if it - it might be funny for a kid who enjoys puns, but for some kids, it is infuriating because the adult, in trying to play a game, seems to the child to be baiting him and trying to upset him. So the child obliges by getting upset. It can take so long to explain it to the transgressing adult, that often I don't try, I just limit their contact with my child as much as I can.
And it's not just the fully Autism Spectrum Disorders (ASD) kids who find this a problem - when easy child 2/difficult child 2 was about 6 years old, she was reading a story to her 4 yo cousin. They were snuggled in cousin's bed under the blankets. Cousin's mum came in and said, "Get out of that bed and pull the covers up! You are clothed, it is the middle of the day. Do not climb under the blankets!"
She then left the room. easy child 2/difficult child 2 made the bed and the two girls got back ON the bed and easy child 2/difficult child 2 continued to read to her cousin. Aunty came back into the room. "You are a naughty girl, you are deliberately being defiant! I told you to get out of the bedroom! Beds are for sleeping in, not reading books. Go out to the couch, that's where you should be. And don't try to find a way to be disobedient again!"
She then came and tackled me about it. I said, "You told them to get out from under the blankets. They did that. You didn't say wanted them out of the room." With small children, as well as some children with social deficits or communication issues, you MUST be specific and concise.
This is typical of the literal-mindedness of these kids. The ADHD kids as well as the Autism Spectrum Disorders (ASD) kids. And some others.
Of course, I was told that no wonder my child was learning to split hairs and be defiant, if I always made excuses like that. But over the years I have watched this woman parent her kids - she doesn't know how to be consistent. But her girls have turned out okay, because they are primarily PCs. She has some spectacular fights with her oldest because of her inconsistency, I stay out of it and do not try to advise because I know my advice is not wanted. I also know nothing I say will make any difference.
All I'm saying - some kids, for whatever reason, have a very neat, intelligent, ordered and logical mind. But if they lack sufficient social understanding (and social rules can be very complex) they will be learning the right way to behave by imitating their role models. They will also apply that ordered and logical mind to their understanding of the rules, as they see them. They also can have a much greater capacity for stubbornness than any parent I know. It can become the irresistible force meeting the immovable object. When it becomes a battle of wills, you will find you have lost the war. These kids are STRONG!
It works better if you lead and deflect, rather than pull the "I am the adult, you are the child" routine. I found when I became my son's facilitator and stopped trying to be the authority figure, I ended up being given more authority - by him. And once he has given me authority over him, I can do far more. Over time he is learning to accept authority over him, from others. Such as teachers, other adults in authority. Of course he has always respected obvious authority figures - those in uniform are publicly branded as having authority where their uniform gives it to them.
I like what you say that he is contrite when he calms down. That is a good time to try to teach him, and it sounds like you are doing that. He sounds like a good kid at heart, but with perhaps impulse control issues as well as some social deficits, in some areas. They do learn and adapt, but often take longer and need to learn social rules a different way.
A lot of this is conjecture - without a diagnosis we can't do much else. It must be so frustrating for you to not be able to get a better idea of what is going on.
difficult child 3 is better than that now, but we've had to learn a different way of managing him. We did a good job with easy child - she is amazing. So it's not us. But the parenting methods we used for easy child, were absolutely the worst we could do with difficult child 3.
So I am not meaning to seem critical of you. Just saying - somehow in this kid's head, things are not working as they should, in some small areas. And you've been trying to be the best parent you can be, for years. However, something is still not right and your efforts are just not working. That is not parental neglect in any way. If anyone says it is, they are talking out the other end. But if you continue to try what is not working and expect a different result - that is the definition of insanity. It will also drive you nuts! Again, I speak from personal experience, I did what you have been doing. Now, a lot of what you are doing is right. But somewhere in there, something isn't, not for this child. There is something else going on, and it seems to me you are trying to control and discipline some aspect of him that cannot be controlled or disciplined, and the result is extreme frustration and anxiety in him, which is ramping up the problem. In an otherwise very bright child, it is very difficult to reconcile that there could be something in him like this. It makes it even more difficult to identify the specific area causing the problem.
A diagnosis would help a great deal, you already know that. I do 'get' the expense side of things. I worry though, that you have people who want to medicate him, but still won't give a diagnosis. I do not think medications should be given without a diagnosis. Even if they just say, "ADHD" which you can have with other things, and the medications are tried and they help - that pretty much confirms the ADHD. But it may not be the whole picture. However, it would be a start.
You've had a really rough time by the sounds of it.
Marg