When you say "sexual act", did it involve coercion of the younger child? Do you think it was an attempt to get sexual gratification in some way? Or was it perhaps curiosity>
I remember being on the edge of puberty and "comparing notes" with a friend of mine. I was a total innocent, my mother chose to tell me nothing other than the basic self-care information. I think she felt that by doing so, she would protect me from my own developing sexuality. She was so wrong! Hormones will out, and by telling me nothing, my mother gave me few hangups. Other than "no premarital sex" - but I had little idea of what sex actually involved.
I remember stories (told by friends and others I've talked to over the years) of their own curiosity in early adolescence. Boys especially will be curious about one another's genitals, wondering if others look the same. If what you caught was simply exploration and curiosity, you do need to emphasise that it is not socially acceptable. Also, if the 5 yo felt uncomfortable and didn't like it, the older one has to learn that it is not right to upset his little brother in such an intimate way. And little brother has to learn that it is OK to feel uncomfortable, such feelings are telling him that what is happening is not right, and he should tell someone so they can make it stop.
If what happened was a bit more involved, be aware tat your 5 yo is probably not the only child who has encountered your older son in tis way. But before you get too angry with your older son, try to find out where this has come from. My eldest easy child was sexually abused by another kid at school, who was 7 years old. What can a 7 yo do to a 5 yo? Enough to have the girl totally terrorised for two years before she told me, and for it to have long-reaching emotional consequences for her. We don't know what happened physically. But emotionally, this boy forced her, threatened her and told her it would be nasty when he had sex with her because sex hurts.
The really upsetting thing about it is, I'm now certain that the 7 yo boy was himself a victim of sexual abuse (how else would he know it hurts?) and was acting out his own experience, on a younger and more vulnerable victim than himself in order to regain the power he felt he had lost. My daughter's attacker needed treatment and rescuing, not punishment. Unfortunately, he was out of reach by the time she told us.
I've also heard of cases at our local elementary school, of 5 yo girls being attacked by older boys. Because the girls were so young they were unable to identify their attackers (had never seen them before) and the school did nothing. And so the attacks can continue.
Kids who attack like this often don't know what is right or wrong, especially if they themselves have been molested. The first thing that needs to happen, is education and counselling. They also need to be reassured that nobody deserves to be controlled and made to do something they feel yuk about in this way. Neither child. From there, it depends on what comes out in the process.
Hang in there, go gently but firmly. Get professional support in this, so you can move forward in a healthy way.
Marg
I am a mother of 3 boys and 1 daughter. My oldest son is 14. My other children are daughter-age 12, DS age 9 and DS age 5. I just found out that my 14 yo attempted to engage DS age 5 in a sexual act. DS refused and went and told dad right away...thank goodness. 14 yo DS lied at first, but then admitted to what he was trying to do. He has always shown signs of ODD and ADHD. The past 2 years have been really great though, and I thought we were through the thick of it. We have had periods of him looking at porn at home (but then we did parental controls..) then looking at porn at other people's homes. 
